tech-tips on takac.dev

Docker: How to solve the trouble of SSH tunnel connection?

Thu, 16 Dec 2021 23:00:00 +0900

I wanted to connect to some external site via my server in AWS because the site has IP address filters as security. However my apartment doesn't have a static IP address so I came up with an idea connecting to the site via SSH tunnel and if it runs as a container in docker-compose, it’s very easy to execute by the command “docker-compose up -d.” But there was a pitfall when digging an SSH tunnel on docker. Let me share what the trap was and how to solve it.

The command which I executed on the CLI was like this.

1ssh -4f -NL 20000:hogeadmin.work:443 takashi@myadmin.server -p 10023

So, I made a simple container with the following Dockerfile to execute ssh client.
[ssh-tunnel/Dockerfile]

 1FROM alpine:latest
 2
 3RUN set -x \
 4    && apk update \
 5    && apk upgrade \
 6    && apk add --no-cache \
 7            openssh-client \
 8            ca-certificates \
 9            bash \
10            bind-tools
11
12EXPOSE 20000-20010
13
14ENTRYPOINT [ "/docker-entrypoint.sh" ]

And, I made a simple script as docker-entrypoint.sh.
[ssh-tunnel/docker-entrypoint.sh]

 1#!/bin/bash
 2REMOTE_HOST='x.x.x.x'
 3REMOTE_USER='user-name'
 4REMOTE_PORT='10023'
 5
 6connect_ssh_tunnel(){
 7    local local_port=$1; shift
 8    local remote_port=$1; shift
 9    local fqdn=$1; shift
10
11    ssh -4f -p ${REMOTE_PORT} -NL ${local_port}:${fqdn}:${remote_port} ${REMOTE_USER}@${REMOTE_HOST}
12}
13
14connect_ssh_tunnel '20000' '443' hogeadmin.work'
15
16# To keep running
17tail -f /dev/null

And then, I made a docker-compose.yml file like this.

 1version: '3.4'
 2services:
 3  ssh-tunnel:
 4    build:
 5      context: ./docker/ssh-tunnel
 6    restart: always
 7    container_name: ssh-tunnel
 8    ports:
 9      - 443:20000
10    volumes:
11      - ./docker/ssh-tunnel/docker-entrypoint.sh:/docker-entrypoint.sh:ro
12      - ~/.ssh:/root/.ssh:ro

So, I run the command “docker-compose up -d” to run the container. And then, I tried to connect via curl command with the “--resolve” option.

1$ curl --resolve hogeadmin.work:443:127.0.0.1 https://hogeadmin.work/
2curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to hogeadmin.work

I was confused because it’s just a simple thing but I got the above error. So, I tried to connect via the telnet command to check the port is alive or not.

1$ telnet localhost 443
2Trying ::1...
3Connected to localhost.
4Escape character is '^]'.
5Connection closed by foreign host.

The connection was closed quickly. So, I tried to do the same thing in the container.

 1$ docker exec -it ssh-tunnel bash
 2
 3bash-5.1# apk update && apk add -f curl
 4fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
 5fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
 6v3.14.3-58-g7fc21b9dfb [https://dl-cdn.alpinelinux.org/alpine/v3.14/main]
 7v3.14.3-57-g005638434d [https://dl-cdn.alpinelinux.org/alpine/v3.14/community]
 8OK: 14942 distinct packages available
 9OK: 21 MiB in 40 packages
10
11bash-5.1# curl localhost:20000
12<html>
13<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
14<body>
15<center><h1>400 Bad Request</h1></center>
16<center>The plain HTTP request was sent to HTTPS port</center>
17</body>
18</html>

I got the HTTP response as the above. So, the SSH tunnel in itself works correctly but it’s not able to connect via the mapped port which is defined in the docker-compose.yml file. So, I searched about the ssh client command and I realized that the “-L” option has the optional parameter called “bind_address.” So, I add "0.0.0.0" as bind address to the command in the script like this.

1connect_ssh_tunnel(){
2   local local_port=$1; shift
3   local remote_port=$1; shift
4   local fqdn=$1; shift
5 
6   ssh -4f -p ${REMOTE_PORT} -NL 0.0.0.0:${local_port}:${fqdn}:${remote_port} ${REMOTE_USER}@${REMOTE_HOST}
7}

And then, I restarted the docker-compose and tried again.

 1$ curl --resolve hogeadmin.work:443:127.0.0.1 https://hogeadmin.work/
 2<!DOCTYPE HTML>
 3<html lang="ja">
 4    <head>
 5        <meta charset="utf-8">
 6        </meta>
 7    </head>
 8    <body>
 9        Forbidden
10    </body>
11</html>

Finally, I got the HTTP response so it works as I expected. However this way is simple way so it has some problems as follows.

It’s impossible to get access to some multiple sites at the same time
It will be a problem when the 443/tcp port is already used in other purpose

But I have a solution that is a little bit complicated so I’ll share the way in another article in this blog in the near future.

Docker: How to install packages to busybox image?

Sat, 07 Aug 2021 23:00:00 +0900

Busybox image is very useful to execute a simple binary on Docker because the image size is quite small. However there's no commands like apt-get or yum to install additional packages. So, if it's necessary to do that, using Alpine Linux image is the best choice instead of Busybox image . Let me share why is it and what we should do when it's necessary.

First, what's "busybox" exactly? According to the official site , it said "BusyBox combines tiny versions of many common UNIX utilities into a single small executable." And also they said it's "The Swiss Army Knife of Embedded Linux." It means that it's a "tool" not a "distribution." On the other hand, Alpine Linux's official site said "Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox ." And also, as you know, Alpine Linux has the package management command "apk." So, it means the following.

1Alpine Linux = BusyBox + musl libc + apk

So, don't try to install some packages to Busybox image . When you want to install something as a package, you should use Alpine Linux image instead of Busybox image . Of course, you can also use other images like Ubuntu, Debian, CentOS and so on but you should accept that its image size is going to be bigger than Alpine Linux image .

Prometheus: How to solve blackbox exporter icmp (ping) failure

Wed, 28 Jul 2021 23:00:00 +0900

I set up my blackbox exporter to monitor some network appliances via ICMP but its RTT was always zero. I'll share how to investigate this problem and solve it.

First I added the following module setting to blackbox.yml because I've never used ICMP monitoring before.

1modules:
2    icmp:
3        prober: icmp
4        timeout: 5s

Then, I added the following setting to the Prometheus configuration file.

1  - job_name: 'icmp_check'
2    metrics_path: /probe
3    params:
4      module: [icmp]
5    static_configs:
6      - targets:
7        - 'target01.hoge'
8        - 'target02.hoge'

I restarted the Prometheus and the blackbox exporter and Prometheus got started to monitor the above two network appliances. Then, I searched its result by the following query on Prometheus's web UI.

1probe_icmp_duration_seconds{job="icmp_check", phase="rtt"}

However, the second one's value had not been captured correctly even though the first one was correct. Actually the 'rtt' was always zero. So, I tried to execute the ping command in the blackbox-exporter's container but the result was no problem. I was not sure why it happened so I decided to add the debug flag to the blackbox-exporter in my docker-compose.yml file like this.

 1 blackbox-exporter:
 2  container_name: blackbox-exporter 
 3  build: ./blackbox_exporter
 4  restart: always
 5  ports:
 6   - "9115:9115"
 7  command:
 8   - "--log.level=debug"
 9  sysctls:
10    net.ipv4.ping_group_range: "0 2147483647"

After restarting the blackbox-exporter, I got the following logs.

1ts=2021-07-01T03:00:00.890Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Beginning probe" probe=icmp timeout_seconds=5
2ts=2021-07-01T03:00:00.890Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolving target address" ip_protocol=ip6
3ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolved target address" ip=xxxx:xxxx:xxxx:xxxx::1
4ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating socket"
5ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating ICMP packet" seq=57536 id=40265
6ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Writing out packet"
7ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Error writing to socket" err="write udp [::]:560->[xxxx:xxxx:xxxx:xxxx::1]:0: sendto: cannot assign requested address"
8ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Probe failed" duration_seconds=0.001720556
9ts=2021-07-01T03:07:19.099Z

And I realized that the second one's FQDN was solved as an IPv6 address. It caused the error because Docker uses the IPv4 only as default. So, it couldn't create a socket to send ICMP packets. That's why I added the following setting to the blackbox.yml file to use IPv4 preferably.

1modules:
2    icmp:
3        prober: icmp
4        timeout: 5s
5        icmp:
6            preferred_ip_protocol: "ip4"

Then, I restarted the blackbox-exporter again and checked the logs.

1ts=2021-07-01T03:16:27.891Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Beginning probe" probe=icmp timeout_seconds=5
2ts=2021-07-01T03:16:27.891Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolving target address" ip_protocol=ip4
3ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolved target address" ip=xxx.xxx.xxx.xxx
4ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating socket"
5ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating ICMP packet" seq=39633 id=48931
6ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Writing out packet"
7ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Waiting for reply packets"
8ts=2021-07-01T03:16:27.997Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Found matching reply packet"
9ts=2021-07-01T03:16:27.997Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Probe succeeded" duration_seconds=0.106222366

The exporter used the IPv4 address as I expected. So, I searched its result by the following query on the web UI again.

1probe_icmp_duration_seconds{job="icmp_check", phase="rtt"}

Finally, the value has been captured correctly. Last, don't forget to turn off the debug log setting for blackbox-exporter to avoid filling up the disk.

1 blackbox-exporter:
2  container_name: blackbox-exporter 
3  build: ./blackbox_exporter
4  restart: always
5  ports:
6   - "9115:9115"
7  # command:
8  #  - "--log.level=debug"

Fluentd: Warning on parser filter: nested repeat operator '+' and '' was replaced with '' in regular expression

Wed, 19 May 2021 00:35:00 +0900

I needed to add parser settings for CoreDNS working on my Kubernetes cluster for troubleshooting but I got an error "nested repeat operator '+' and '' was replaced with '' in regular expression." I had no idea why it happened because the additional regular expression was valid. The cause of the error was my mistake so let me share what’s the cause of?

First, I added log setting to the CoreDNS configuration like this.

 1$ kubectl get configmaps -n kube-system coredns -o yaml
 2apiVersion: v1
 3data:
 4  Corefile: |
 5    .:53 {
 6        log
 7
 8        errors
 9        health {
10           lameduck 5s
11        }
12        ready
13     :
14     :

And then, I restarted the pod and got logs like this.

1[INFO] 172.31.57.199:54169 - 1916 "A IN dynamodb.us-west-2.amazonaws.com. udp 50 false 512" NOERROR qr,rd,ra 98 0.000921506s

So, I create a regular expression string and test it on https://fluentular.herokuapp.com/. Then, added the following filter to the Fluentd configuration.

 1<filter **>
 2    @type parser
 3    key_name log
 4    reserve_data true
 5    replace_invalid_sequence true
 6    <parse>
 7        @type multi_format
 8        <pattern>
 9            format regexp
10            expression /^\[(?<severity>\S+)\] (?<remote>\d+\.\d+\.\d+\.\d+):(?<port>\d+) +- (?<id>\d+) +*"(?<type>\S+) +(?<class>\S+) +(?<name>[^ ]+)\. +(?<proto>\S+) +(?<size>\d+) +(?<do>\S+) +(?<bufsize>\d+)" +(?<rcode>\S+) +(?<rflags>[^ ]+) +(?<rsize>\d+) +(?<duration>\d+\.\d+)s$/
11        </pattern>
12     :
13     :

After reloading the fluentd pod, then the following error appeared.

12021-05-17 08:57:06 +0000 [info]: adding filter pattern="**" type="parser"
2/usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/config/types.rb:92: warning: nested repeat operator '+' and '*' was replaced with '*' in regular expression

So, I deleted the expression and then I paste each element of the regular expression from the beginning one by one and then I found out the cause of the problem is "(?\d+) +*"(?\S+)." As you see, there is "+*" as my mistake. So, I modify the part like this.

1^\[(?<severity>\S+)\] (?<remote>\d+\.\d+\.\d+\.\d+):(?<port>\d+) +- (?<id>\d+) +"(?<type>\S+) +(?<class>\S+) +(?<name>[^ ]+)\. +(?<proto>\S+) +(?<size>\d+) +(?<do>\S+) +(?<bufsize>\d+)" +(?<rcode>\S+) +(?<rflags>[^ ]+) +(?<rsize>\d+) +(?<duration>\d+\.\d+)s$

And then, I restarted the pod and the errors didn’t appear.

PHP: Fatal error when build docker image with composer

Fri, 07 May 2021 00:25:00 +0900

I saw this news about the serious vulnerability in composer which is PHP package management software.

https://www.bankinfosecurity.com/php-composer-flaw-that-could-affect-millions-sites-patched-a-16523

So, I wanted to update composer in my Dockerfile but I realized that I already specified the version as follows.

1curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer --version=1.10.16

Because it was a workaround for an error related to hirak/prestissimo which I did before. So, I deleted it and built the Docker image again but I got a different error as follows.

1#14 281.5  122/122 [============================] 100%  - Installing kylekatarnls/update-helper (1.2.0): Extracting archive
2#14 281.5 
3#14 281.5 Fatal error: Class UpdateHelper\ComposerPlugin contains 2 abstract methods and must therefore be declared abstract or implement the remaining methods (Composer\Plugin\PluginInterface::deactivate, Composer\Plugin\PluginInterface::uninstall) in …

It looks some problem about kylekatarnls/update-helper library. So, I searched about the library and I found a clue on GitHub.

https://github.com/kylekatarnls/update-helper/issues/7

So, I deleted composer.lock file and built the image again. Then the problem was solved. The above page said I need to delete the ‘vendor’ directory but it’s not necessary for docker build because it will be generated newly.

Python: Got AttributeError when accessing App Store Connect API using appstoreconnect library

Thu, 06 May 2021 23:55:00 +0900

I created a small Python program to get data via the App Store Connect API for report automation with the appstoreconnect library. After update recently, I got the following error.

 1Traceback (most recent call last):
 2  File "/app/./main.py", line 88, in <module>
 3    APP_STORECONNECT_API.execute()
 4  File "/app/app_store_connect.py", line 176, in execute
 5    api = Api(self.API_KEY, self.CERT_FILE, self.ISSUER_ID)
 6  File "/usr/local/lib/python3.9/site-packages/appstoreconnect/api.py", line 51, in __init__
 7    token = self.token  # generate first token
 8  File "/usr/local/lib/python3.9/site-packages/appstoreconnect/api.py", line 269, in token
 9    self._token = self._generate_token()
10  File "/usr/local/lib/python3.9/site-packages/appstoreconnect/api.py", line 64, in _generate_token
11    return jwt.encode({'iss': self.issuer_id, 'exp': exp, 'aud': 'appstoreconnect-v1'}, key,
12AttributeError: 'str' object has no attribute 'decode'

It depends on the above error, the problem happened here.

https://github.com/Ponytech/appstoreconnectapi/blob/0916ac75790b13cf6c89617209a1a6c2db1f16d1/appstoreconnect/api.py#L79-L80

And the author already realized about this issue.

https://github.com/Ponytech/appstoreconnectapi/issues/37

He said the cause of the problem is the type of return value was changed in the JWT library from byte to string since 2.0. So, I thought I should specify a version of JWT which is the latest before 2.0. And then, I checked the version history here.

https://pypi.org/project/PyJWT/#history

The latest 1.x version is 1.7.1 so I specified it explicitly when executing the pip command in my Dockerfile like this.

1pip install PyJWT==1.7.1

The problem was solved. So, this is one of the workarounds for now about this issue.

Example of Selenium with Python on Docker with latest FireFox

Wed, 28 Apr 2021 00:24:00 +0900

This is an example of how to run the Selenium with Python on Docker with the latest FireFox and its geckodriver.

The official Python docker image is based on Debian 10 (Buster)

1$ docker run -i --rm python:3.9 cat /etc/os-release | grep VERSION=
2VERSION="10 (buster)"

In this image, the installable FireFox package by the default registered repository is the ESR version which stands for Extended Support Release. It’s like LTS on Ubuntu and this version is not up-to-date. At the time of this writing, the latest version is 87 but the ESR is 78.10.

1$ docker run -it --rm python:3.9 sh -c "apt update; apt search firefox | grep firefox-esr/stable"
2firefox-esr/stable 78.10.0esr-1~deb10u1 amd64

And also, geckodriver which is necessary to use by Selenium is not included. That’s why I needed to install manually. Here is a Dockerfile example.

 1FROM python:3.9
 2 
 3ENV DEBIAN_FRONTEND noninteractive
 4ENV GECKODRIVER_VER v0.29.0
 5ENV FIREFOX_VER 87.0
 6 
 7RUN set -x \
 8   && apt update \
 9   && apt upgrade -y \
10   && apt install -y \
11       firefox-esr \
12   && pip install  \
13       requests \
14       selenium \
15 
16# Add latest FireFox
17RUN set -x \
18   && apt install -y \
19       libx11-xcb1 \
20       libdbus-glib-1-2 \
21   && curl -sSLO https://download-installer.cdn.mozilla.net/pub/firefox/releases/${FIREFOX_VER}/linux-x86_64/en-US/firefox-${FIREFOX_VER}.tar.bz2 \
22   && tar -jxf firefox-* \
23   && mv firefox /opt/ \
24   && chmod 755 /opt/firefox \
25   && chmod 755 /opt/firefox/firefox
26  
27# Add geckodriver
28RUN set -x \
29   && curl -sSLO https://github.com/mozilla/geckodriver/releases/download/${GECKODRIVER_VER}/geckodriver-${GECKODRIVER_VER}-linux64.tar.gz \
30   && tar zxf geckodriver-*.tar.gz \
31   && mv geckodriver /usr/bin/
32 
33COPY ./app /app
34 
35WORKDIR /app
36 
37CMD python ./main.py

And this is an example using Selenium in Python code. [app/scraper.py]

 1from selenium import webdriver
 2from selenium.webdriver.firefox.options import Options
 3from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
 4import time
 5 
 6# FireFox binary path (Must be absolute path)
 7FIREFOX_BINARY = FirefoxBinary('/opt/firefox/firefox')
 8 
 9# FireFox PROFILE
10PROFILE = webdriver.FirefoxProfile()
11PROFILE.set_preference("browser.cache.disk.enable", False)
12PROFILE.set_preference("browser.cache.memory.enable", False)
13PROFILE.set_preference("browser.cache.offline.enable", False)
14PROFILE.set_preference("network.http.use-cache", False)
15PROFILE.set_preference("general.useragent.override","Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0")
16 
17# FireFox Options
18FIREFOX_OPTS = Options()
19FIREFOX_OPTS.log.level = "trace"    # Debug
20FIREFOX_OPTS.headless = True
21GECKODRIVER_LOG = '/geckodriver.log'
22 
23class Scraper:
24   def __init__(self):
25	ff_opt = {
26		firefox_binary=FIREFOX_BINARY,
27		firefox_profile=PROFILE,
28		options=FIREFOX_OPTS,
29		service_log_path=GECKODRIVER_LOG
30	}
31       self.DRIVER = webdriver.Firefox(**ff_opt)
32 
33   def scrape(self, link):
34       try:
35           self.DRIVER.get(link)
36           time.sleep(5) # just in case
37           html = self.DRIVER.page_source
38 
39           return html
40 
41       except Exception as e:
42           print(e)

Execute the above class like this. [app/main.py]

1from scraper import Scraper
2 
3pretty_html = SCRAPER.scrape(link).prettify()

One more tips, on ubuntu:20.04 docker image it’s possible to install the latest FireFox package, not ESR. So, another way is setting up Python environment based on this image.

1$ docker run -it --rm ubuntu:20.04 sh -c "apt update; apt search firefox | grep ^firefox/"
2firefox/focal-updates,focal-security 88.0+build2-0ubuntu0.20.04.1 amd64

Reference

https://www.linuxuprising.com/2019/12/how-to-install-latest-firefox-non-esr.html

Ruby: AWS SAM build error with mongoid package

Fri, 05 Feb 2021 23:59:00 +0900

Same as my last post , I’m not a Ruby developer but I had an opportunity to help move some Ruby applications from EC2 to Lambda on AWS. At the time, I had a problem when I built a Ruby application with AWS SAM framework. Let me share why it happened and how I fixed it.

I executed the build command "sam build‘ for the Ruby application but it failed with the following error message.

1Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "RubyBundlerBuilder:CopySource - [Errno 2] No such file or directory: '/tmp/samcli/source/vendor/bundle/ruby/2.7.0/gems/mongo-2.14.0/spec/support/ocsp'"}}
2
3Build Failed
4Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 72694, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1, 'requestId': 'c9a32492-242a-460b-8633-9c2b8c4b7650', 'installationId': 'a15de333-32e0-4651-b78a-c0843ab41c31', 'sessionId': '970cf38b-8715-42bb-b726-c664597c56ab', 'executionEnvironment': 'CLI', 'pyversion': '3.8.7', 'samcliVersion': '1.15.0'}}]}

It was weird for me because all package files were already installed via "bundle install" command. However, the above error said "No such file or directory" and the file is the following.

1'/tmp/samcli/source/vendor/bundle/ruby/2.7.0/gems/mongo-2.14.0/spec/support/ocsp'

In my Gemfile, I specified 'mongoid' and when I executed the "bundle install" command, the ‘mongo’ package was also installed because of the dependency. The automatically installed version was "2.14.0" which is the latest version and I checked the above file path in its GitHub repository.

https://github.com/mongodb/mongo-ruby-driver/blob/master/spec/support/ocsp

And then, I realized that the file is a symbolic link to the following path.

1../../.mod/drivers-evergreen-tools/.evergreen/ocsp

And the .mod directory in the above path and "drivers-evergreen-tools" is a symbolic link to the other repository called drivers-evergreen-tools . That’s why it's become a broken link in the installed source code directory.

So, I checked the commit log in the mongo package repository and the file was added on 9 Sep 2020. And then, I saw the list of the mongo package’s version on the gems site. And it said as follows.

12.14.0 - December 01, 2020 (886KB)
22.14.0.rc1 - October 09, 2020 (883KB)
32.13.2 - December 01, 2020 (862KB)
42.13.1 - October 09, 2020 (855KB)
52.13.0 - July 30, 2020 (855KB)

So first, I deleted the installed directory and modified the version to 2.13.2 in the Gemfile as follows. And then, install the package again via "bundle install" command.

1gem 'mongo', '~> 2.13.2'

Luckily, the broken link had disappeared and succeeded to build by "sam build" command.

So, I’m not sure what’s OCSP exactly and why they put the link in the repository. However, I hope it will be helpful for somebody who has the same problems.

Ruby: bundle install hangs indefinitely with high CPU usage

Sun, 31 Jan 2021 00:08:05 +0900

Actually, I’m not a Ruby developer but I had an opportunity to help move some Ruby applications from EC2 to Lambda on AWS. At that time, I had an issue when I execute "bundle install" inside the container to build with the SAM framework. The problem was the command didn’t finish indefinitely. I searched about it and found some people who had the same issue but there was no way to work around it. I think changing or deleting versions will solve the problem. Let me share how to do that.

First, comment out all lines in the Gemfile, and then enable a package one by one. If you find the package which is the cause of the hanging, try the following way.

If it’s already specified some version, delete it.
Specify its latest version explicitly.
Specify the previous version of the latest one or the one after next

In my case, "activesupport" was the cause of the issue. Before I modified the Gemfile, it did have specified versions explicitly and installed "6.0.3.4" automatically by bundle. However, I checked rubygems.org and the latest version was "6.1.1" so I wrote it down like this and the problem was solved.

1gem 'activesupport', '~> 6.1.1'

I’m not sure this way will solve all the same issues but I hope it will be helpful for somebody who has the same problems.

Docker: What should I do when getting "503 service unavailable" error after executing docker pull command?

Sat, 26 Dec 2020 22:04:00 +0900

Getting "503 service unavailable" error after executing "docker pull" command is a very rare situation but there was in the past. So, it’s worth knowing for using docker. Most of the time, this is Docker Hub’ server side problem not client one. Once the error happens, we should visit their web site called Docker system status real-time view before struggling. When the error happened, may be "Docker Hub Registry" is in some trouble status.

Docker: How to solve Apache 403 Forbidden error?

Fri, 25 Dec 2020 01:28:00 +0900

It's a popular problem getting "403 Forbidden" errors from Apache. Even though it's working on the docker, it's the same but a little bit harder to investigate why it’s happened. However, most of the time, the cause of this problem is simple. Let’s take a look at some samples.

First, let me show you the correct way. Here is the vhost configuration file as vhost.conf.

1<VirtualHost *:80>
2    DocumentRoot /var/www/html
3</VirtualHost

And make a directory and put a text file for verifying.

1mkdir app/web
2echo "Hoge" > app/web/index.html

And make a docker-compose.yml file like this.

 1version: '3'
 2services:
 3  app:
 4    container_name: my-app
 5    image: php:7.4-apache
 6    ports:
 7      - 8080:80
 8    volumes:
 9      - ./app/web:/var/www/html
10      - ./vhost.conf:/etc/apache2/sites-enabled/000-default.conf

Then, it’s ready for running. Execute it like this.

1docker-compose up

And open a new terminal and execute curl command for verifying.

1$ curl localhost:8080
2Hoge

This is what I expected. It’s totally no problem. But sometimes "403 Forbidden" errors appear. How it happens? Most of the time, the cause of the error is modifying document root directory. Here is a sample.

Modify the document root setting in vhost.conf

1<VirtualHost *:80>
2    DocumentRoot /srv/app
3</VirtualHost>

And modify docker-compose.yml too.

 1version: '3'
 2services:
 3  app:
 4    container_name: my-app
 5    image: php:7.4-apache
 6    ports:
 7      - 8080:80
 8    volumes:
 9      - ./app/web:/srv/app
10      - ./vhost.conf:/etc/apache2/sites-enabled/000-default.conf

However, if I execute curl command the same as the last time, the error will be returned.

 1$ curl localhost:8080
 2<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 3<html><head>
 4<title>403 Forbidden</title>
 5</head><body>
 6<h1>Forbidden</h1>
 7<p>You don't have permission to access this resource.</p>
 8<hr>
 9<address>Apache/2.4.38 (Debian) Server at localhost Port 8080</address>
10</body></html>

The hint is Apache's default settings.

 1$ docker run -it --rm --entrypoint sh php:7.4-apache -c "cat /etc/apache2/apache2.conf"
 2  :
 3  :
 4<Directory />
 5        Options FollowSymLinks
 6        AllowOverride None
 7        Require all denied
 8</Directory>
 9
10<Directory /usr/share>
11        AllowOverride None
12        Require all granted
13</Directory>
14
15<Directory /var/www/>
16        Options Indexes FollowSymLinks
17        AllowOverride None
18        Require all granted
19</Directory>
20  :
21  :

As you see above, the first setting is set "all denied" for the root directory. And then, "/usr/share" and "/var/www/" is granted. So, even though I modify the directory path in the vhost.conf and docker-compose.yml file, the modified directory is not granted. So, the response will be the error.

So, how to fix it when I need to modify the document root directory? It’s necessary to add a grant setting like this.

1<VirtualHost *:80>
2    DocumentRoot /srv/app/web
3
4    <Directory "/srv/app/web">
5        AllowOverride all
6        Require all granted
7    </Directory>
8</VirtualHost>

The error will disappear.

And also, mistakes of the mount point is also a popular cause of some problems. For example, confusing "sites-available" and "sites-enabled" directories. In the traditional way, add the configuration file to "sites-available" first, and then execute "a2ensite" command to make it enabled and then restart the Apache process with graceful option. However, when executing the Apache on docker, no need to do that. Just mount it to 000-default.conf file under the "sites-enabled." If it’s mounted under the "sites-available" as follows, it will not be loaded by Apache.

1    volumes:
2      - ./app/web:/srv/app
3      - ./vhost.conf:/etc/apache2/sites-available/000-default.conf

Docker: Run apache as non-root user based on the official image

Tue, 22 Dec 2020 23:40:00 +0900

Many docker images execute its main process as root but it’s not good for security reasons and it depends on Kubernetes cluster, Pod Security Policies is applied to avoid running containers as root. So, let’s take a look at how to run Apache Web Server as a non-root user.

Before trying, here is running the origin image without modifying.

 1$ docker run -dt \
 2--name my-app \
 3--rm httpd:2.4
 4
 5$ docker exec -it my-app sh -c "apt update; apt upgrade -y; apt install -y procps; ps -aux"
 6
 7<installing>
 8
 9USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
10root         1  0.3  0.0   5940  4388 pts/0    Ss+  03:48   0:00 httpd -DFOREGROUND
11daemon       8  0.0  0.1 1931504 11660 pts/0   Sl+  03:48   0:00 httpd -DFOREGROUND
12daemon       9  0.0  0.1 1931504 11660 pts/0   Sl+  03:48   0:00 httpd -DFOREGROUND
13daemon      10  0.0  0.1 1931504 11660 pts/0   Sl+  03:48   0:00 httpd -DFOREGROUND
14root        92  0.1  0.0   2388   696 pts/1    Ss+  03:48   0:00 sh -c apt update; apt upgrade -y; 
15apt install -y procps; ps -aux
16root       409  0.0  0.0   7640  2676 pts/1    R+   03:48   0:00 ps -aux
17
18$ docker stop my-app

As you see, the first process is executed as root and other sub processes run as "daemon" user. When executing the "docker run" command, all I need to do to run the container as some specified user is adding the "-u" option. However, if I add the option without any modification, the container cannot use port 80 because it’s permitted to root only as default. That’s why it’s necessary to use the "setcap" command to modify the linux capability. So, I created the following Dockerfile.

1FROM httpd:2.4
2
3RUN apt update \
4    && apt upgrade -y \
5    && apt install -y libcap2-bin procps \
6    && setcap 'cap_net_bind_service=+ep' /usr/local/apache2/bin/httpd \
7    && chown www-data:www-data /usr/local/apache2/logs
8
9USER www-data

In the above file, "procps" package is unnecessary for production. It’s just for verifying. And then, build the image.

1docker build -t non-root-apache .

After the building process, run a container.

1docker run -it --rm \
2--name my-apache-app \
3-p 8888:80 \
4-v `pwd`/www:/usr/local/apache2/htdocs \
5non-root-apache

After the container is up, open another terminal and test it via curl command.

1$ curl localhost:8888/test.txt
2hoge

The Apache’s log is also like this on the terminal as stdout.

1172.17.0.1 - - [22/Dec/2020:08:53:03 +0000] "GET /test.txt HTTP/1.1" 200 5

Just in case, check the result of ps command again.

1$ docker exec -it my-apache-app ps -aux
2USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
3www-data     1  0.0  0.0   5940  4244 pts/0    Ss+  08:16   0:00 httpd -DFOREGROUND
4www-data     7  0.0  0.0 1931512 5420 pts/0    Sl+  08:16   0:00 httpd -DFOREGROUND
5www-data     8  0.0  0.1 1931512 7516 pts/0    Sl+  08:16   0:00 httpd -DFOREGROUND
6www-data     9  0.0  0.0 1931560 6060 pts/0    Sl+  08:16   0:00 httpd -DFOREGROUND
7www-data   106  0.0  0.0   7640  2676 pts/1    Rs+  08:54   0:00 ps -aux

As you see, all running processes are executed by the "www-data" user.

AWS: Create external Aurora replication host by CLI

Tue, 08 Dec 2020 22:08:00 +0900

One of the big advantages of using AWS is that it’s able to use reliable managed database services like RDS, Aurora. It’s also very easy to increase or decrease replicas. However sometimes, it’s necessary to create a replica externally to avoid bad effects on the production environment. I found the official document and I created an Aurora’s replica following the document. So, let me share the procedure.

First, it needs to modify the Aurora’s default settings as follows.

Set binlog_format parameter to MIXED
Set binlog_checksum parameter to NONE

Then, it needs to reboot to apply the above parameters. So, it will happen 10 or 20 seconds down time.

After rebooting, it should make sure those parameters are applied correctly.

 1mysql> SHOW VARIABLES LIKE 'binlog_format';
 2+---------------+-------+
 3| Variable_name | Value |
 4+---------------+-------+
 5| binlog_format | MIXED |
 6+---------------+-------+
 71 row in set (0.01 sec)
 8
 9mysql> show binary logs;
10+----------------------------+-----------+
11| Log_name                   | File_size |
12+----------------------------+-----------+
13| mysql-bin-changelog.000001 |       120 |
14| mysql-bin-changelog.000002 |       120 |
15+----------------------------+-----------+
162 rows in set (0.00 sec)

Then, make the retention term of the binlog file longer.

1mysql> CALL mysql.rds_set_configuration('binlog retention hours', 144);
2Query OK, 0 rows affected (0.00 sec)

And, create a replication user.

1mysql> grant replication slave on *.- to 'repl'@'%' identified by 'fuga';
2Query OK, 0 rows affected (0.01 sec)

Set some common variables.

1db_cluster_identifier='temp-db-cluster'
2db_instance_identifier='temp-db-instance'
3db_instance_class='db.t2.small'
4db_engine='aurora-mysql'
5cluster_parameter_group=’my-cluster-param’
6db_parameter_group_name=’my-db-param’
7db_name=’targetdatabase’

Get the latest snapshot ID.

1snapshot_identifier=`aws rds describe-db-cluster-snapshots \
2        | jq -r '.DBClusterSnapshots
3         | sort_by(.SnapshotCreateTime)
4         | reverse
5         | .[0].DBClusterSnapshotIdentifier'`

Restore a new cluster from the above snapshot.

1aws rds restore-db-cluster-from-snapshot \
2--engine                            $db_engine \
3--db-cluster-identifier             $db_cluster_identifier \
4--snapshot-identifier               $snapshot_identifier \
5--db-cluster-parameter-group-name   $cluster_parameter_group \
6--no-deletion-protection

Wait for creating the above process. After it’s completed, add a new instance to the above cluster.

1aws rds create-db-instance \
2--engine                    $db_engine \
3--db-instance-identifier    $db_instance_identifier \
4--db-instance-class         $db_instance_class \
5--db-parameter-group-name   $db_parameter_group_name \
6--db-cluster-identifier     $db_cluster_identifier

Wait for adding the instance process. Then, get the binlog’s position info from its events.

1aws rds describe-events \
2--source-identifier $db_instance_identifier \
3--source-type db-instance \
4| jq -r '.Events[] | .Message | select (. | test("Binlog"))'
5
6# Output is like this
7# Binlog position from crash recovery is mysql-bin-changelog.000007 91466190

Get dump data from the above temporary cluster.

 1MYSQL_PWD=’pass’
 2mysqldump -h $db_endpoint \
 3  --user=user \
 4  --quote-names \
 5  --flush-logs \
 6  --master-data=2 \
 7  --single-transaction \
 8  --hex-blob \
 9  --routines \
10  --triggers \
11  --events \
12  --default-character-set=utf8 \
13  $db_name \
14  > dump_${db_cluster_identifier}.sql

Create a MySQL server out of the Aurora cluster. At this time, I used docker-compose on a server.

 1version: '3.7'
 2services:
 3 local-replica:
 4  image: mysql:5.7
 5  restart: always
 6  ports:
 7   - "13306:3306"
 8  volumes:
 9    - ./mysql:/var/lib/mysql
10    - ./repl.cnf:/etc/mysql/conf.d/repl.cnf
11  env_file:
12   - .env_mysql
13  user: mysql

The above repl.cnf file is like this.

1[mysqld]
2server-id=100
3replicate-ignore-db=mysql

And make the "mysql" directory to be mounted for data.

1mkdir mysql

Then, run the container with the official MySQL image.

1docker-compose up -d

Import the dump file.

1mysql \
2--protocol tcp \
3-P 13306 \
4-u root \
5-p \
6< dump_${db_cluster_identifier}.sql

Login and connect to the above DB and execute CHANGE MASTER command to modify the replication master info with the above binlog position info.

1mysql> CHANGE MASTER TO
2  MASTER_HOST = 'my-master-host',
3  MASTER_PORT = 3306,
4  MASTER_USER = 'repl',
5  MASTER_PASSWORD =’fuga’,
6  MASTER_LOG_FILE = 'mysql-bin-changelog.000007',
7  MASTER_LOG_POS = 91466190;

Then, add grants on the local replica.

1mysql> GRANT SELECT, RELOAD, PROCESS, REFERENCES, INDEX, SHOW DATABASES, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, SHOW VIEW, EVENT, TRIGGER ON *.- TO 'hoge'@'%' IDENTIFIED BY PASSWORD '*AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' WITH GRANT OPTION

Start slave.

 1mysql> start slave;
 2
 3mysql> show slave status\G;
 4**************************- 1. row ***************************
 5              Slave_IO_State: Waiting for master to send event
 6                  Master_Host: my-master-host
 7                  Master_User: repl
 8                  Master_Port: 3306
 9                Connect_Retry: 60
10              Master_Log_File: mysql-bin-changelog.000002
11          Read_Master_Log_Pos: 4656141
12              Relay_Log_File: 80a8d68c1881-relay-bin.000002
13                Relay_Log_Pos: 351215
14        Relay_Master_Log_File: mysql-bin-changelog.000002
15            Slave_IO_Running: Yes
16            Slave_SQL_Running: Yes
17                                :
18                                :
191 row in set (0.00 sec)
20
21ERROR:
22No query specified

The temporary cluster is unnecessary anymore. So, it should be deleted. First, delete the instance.

1aws rds delete-db-instance \
2--db-instance-identifier $db_instance_identifier \
3--skip-final-snapshot

Then, delete the cluster.

1aws rds delete-db-cluster \
2--db-cluster-identifier $db_cluster_identifier \
3--skip-final-snapshot

Docker Hub: Error 429 too many requests

Thu, 19 Nov 2020 14:17:00 +0900

One day, I realized that one of my CodePipeline was failed to build the docker image. I investigated what caused the error. And then, I found out the following error message when executing docker build command.

1ERROR: failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/php/manifests/sha256:7b1cd0d9e1922e96ad3bc9fc44880cf7247f5a22a4badfd25480bf759edf2804: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I remembered the news which I read before that Docker Hub got started to set some limits and I checked the link which is included in the above error.

Understanding Docker Hub Rate Limiting

However, I was confused because I’m working for a very small start-up company and the CodePipeline runs a few times a day so I couldn’t understand why I reached the limit. Then, I came out that it may be "docker login" is necessary because the CodePipeline environment is shared servers so anonymous access was added up together. So, I got a new log-in account for building image and added the following command before docker build command in the buildspec in the CodePipeline and then I solved the problem.

1docker login -u hoge-hoge -p fuga

Kibana query language cheat sheet for web service

Mon, 09 Nov 2020 17:00:00 +0900

I’m using ELK stack for collecting and searching logs from web servers, CDN, load balancers and so on. Let me share my cheat sheet of the Kibana query language I usually use. I’ll update this continually.

Hide empty line

If the logs are not well formatted, some empty line will appear and it’s bothering. So, I hide it by this query.

 1{
 2  "query": {
 3    "bool": {
 4      "must": [
 5        {
 6          "regexp": {
 7            "message.keyword": "\n"
 8          }
 9        }
10      ]
11    }
12  }
13}

Redirected HTTP access

Redirecting from HTTP to HTTPS is a popular way for load balancer settings because of the AOSSL (Always On SSL). It brings a lot of logs of the redirecting but most of the time it’s unnecessary for debugging or investigating issues. So, I hide it by this query.

 1{
 2  "query": {
 3    "bool": {
 4      "must": [
 5        {
 6          "match": {
 7            "cs-protocol.keyword": "http"
 8          }
 9        },
10        {
11          "match": {
12            "sc-status": "301"
13          }
14        }
15      ]
16    }
17  }
18}

Must and Must Not

Sometimes, it’s necessary to combine complicated conditions. For example, the following query will pick up the logs which has "JP" on "geoip_country_code" field and also the status is not 2xx and 3xx.

 1{
 2  "query": {
 3    "bool": {
 4      "must": [
 5        {
 6          "match": {
 7            "geoip_country_code": "JP"
 8          }
 9        }
10      ],
11      "must_not": [
12        {
13          "range": {
14            "status": {
15              "gte": 200,
16              "lt": 400
17            }
18          }
19        }
20      ]
21    }
22  }
23}

Exclude bot logs by UA

There are a bunch of access to public web servers and CDN logs. When debugging, sometimes it’s bothering to search some specified logs. In that situation, I’ll use the following query to filter by the user-agents. Of Course it’s not perfect because there are millions of bots around the world and some wicked bots have fake user-agents but better than nothing.

  1{
  2  "query": {
  3    "bool": {
  4      "should": [
  5        {
  6          "regexp": {
  7            "cs(User-Agent)": ".*bot.*"
  8          }
  9        },
 10        {
 11          "regexp": {
 12            "cs(User-Agent).keyword": ".*spider.*"
 13          }
 14        },
 15        {
 16          "regexp": {
 17            "cs(User-Agent).keyword": ".*google.*"
 18          }
 19        },
 20        {
 21          "regexp": {
 22            "cs(User-Agent).keyword": ".*Google.*"
 23          }
 24        },
 25        {
 26          "regexp": {
 27            "cs(User-Agent).keyword": ".*curl.*"
 28          }
 29        },
 30        {
 31          "regexp": {
 32            "cs(User-Agent).keyword": ".*python.*"
 33          }
 34        },
 35        {
 36          "regexp": {
 37            "cs(User-Agent).keyword": ".*Photon.*"
 38          }
 39        },
 40        {
 41          "regexp": {
 42            "cs(User-Agent).keyword": ".*cortex.*"
 43          }
 44        },
 45        {
 46          "regexp": {
 47            "cs(User-Agent).keyword": ".*crawler.*"
 48          }
 49        },
 50        {
 51          "regexp": {
 52            "cs(User-Agent).keyword": ".*Crawler.*"
 53          }
 54        },
 55        {
 56          "regexp": {
 57            "cs(User-Agent).keyword": ".*daum.*"
 58          }
 59        },
 60        {
 61          "regexp": {
 62            "cs(User-Agent).keyword": ".*Hatena.*"
 63          }
 64        },
 65        {
 66          "regexp": {
 67            "cs(User-Agent).keyword": ".*ltx71.*"
 68          }
 69        },
 70        {
 71          "regexp": {
 72            "cs(User-Agent).keyword": "newspaper/.*"
 73          }
 74        },
 75        {
 76          "regexp": {
 77            "cs(User-Agent).keyword": ".*Qwantify.*"
 78          }
 79        },
 80        {
 81          "regexp": {
 82            "cs(User-Agent).keyword": ".*ubermetrics-technologies.com.*"
 83          }
 84        },
 85        {
 86          "regexp": {
 87            "cs(User-Agent).keyword": "WF%2520search/Nutch-.*"
 88          }
 89        },
 90        {
 91          "regexp": {
 92            "cs(User-Agent).keyword": "Apache-HttpClient.*"
 93          }
 94        },
 95        {
 96          "regexp": {
 97            "cs(User-Agent).keyword": "Mozilla.*Yeti/.*https://naver.me/spd.*"
 98          }
 99        },
100        {
101          "regexp": {
102            "cs(User-Agent).keyword": ".*TrendsmapResolver.*"
103          }
104        },
105        {
106          "regexp": {
107            "cs(User-Agent).keyword": "-"
108          }
109        }
110      ]
111    }
112  }
113}

AWS: CloudFormation example creating DynamoDB table and IAM

Tue, 20 Oct 2020 14:13:00 +0900

Let me share an example of the CloudFormation configuration to create DynamoDB tables and an IAM user with particular policy to use it from some applications.

I’ll create two tables for stage and production with TTL and also I set PAY_PER_REQUEST. So first, create the following YAML file as template.yml.

 1AWSTemplateFormatVersion: "2010-09-09"
 2Description: "DynamoDB and IAM"
 3
 4Resources:
 5  Policy:
 6    Type: 'AWS::IAM::ManagedPolicy'
 7    Properties:
 8      Path: /
 9      PolicyDocument:
10        Version: 2012-10-17
11        Statement:
12          - Effect: Allow
13            Action:
14              - dynamodb:List*
15              - dynamodb:DescribeReservedCapacity*
16              - dynamodb:DescribeLimits
17              - dynamodb:DescribeTimeToLive
18            Resource:
19              - "*"
20
21          - Effect: Allow
22            Action:
23              - dynamodb:DescribeTable
24              - dynamodb:GetItem
25              - dynamodb:Query
26              - dynamodb:Scan
27              - dynamodb:CreateTable
28              - dynamodb:DeleteItem
29              - dynamodb:UpdateItem
30              - dynamodb:UpdateTable
31              - dynamodb:PutItem
32            Resource:
33              - !GetAtt DynamoDBTableUserListProd.Arn
34              - !GetAtt DynamoDBTableUserListStage.Arn
35  User:
36    Type: AWS::IAM::User
37    Properties: 
38      ManagedPolicyArns: 
39        - !Ref Policy
40      UserName: dynamodb-user-list
41      
42  DynamoDBTableUserListProd:
43    Type: AWS::DynamoDB::Table
44    Properties:
45      TableName: "user_list_prod"
46      BillingMode: "PAY_PER_REQUEST"
47      KeySchema:
48        -
49          AttributeName: "user_id"
50          KeyType: "HASH"
51      AttributeDefinitions:
52        -
53          AttributeName: "user_id"
54          AttributeType: "N"
55      ProvisionedThroughput:
56        ReadCapacityUnits: "0"
57        WriteCapacityUnits: "0"
58
59  DynamoDBTableUserListStage:
60    Type: AWS::DynamoDB::Table
61    Properties:
62      TableName: "user_list_stage"
63      BillingMode: "PAY_PER_REQUEST"
64      KeySchema:
65        -
66          AttributeName: "user_id"
67          KeyType: "HASH"
68      AttributeDefinitions:
69        -
70          AttributeName: "user_id"
71          AttributeType: "N"
72      ProvisionedThroughput:
73        ReadCapacityUnits: "0"
74        WriteCapacityUnits: "0"
75      TimeToLiveSpecification:
76        AttributeName: expired_at
77        Enabled: true

And then, create the script as deploy.sh to deploy it.

 1#!/bin/bash
 2STACK_NAME="DynamoDBTableUserList"
 3S3_BUCKET="your_bucket"
 4TEMPLATE_FILE='template.yml'
 5OUTPUT_YAML='output.yml'
 6
 7echo -n "Packaging stack..."
 8aws cloudformation package \
 9--template-file         $TEMPLATE_FILE \
10--s3-bucket             $S3_BUCKET \
11--output-template-file  $OUTPUT_YAML
12echo " Done"
13
14echo -n "Deploying stack... "
15aws cloudformation deploy \
16--template-file     $OUTPUT_YAML \
17--stack-name        $STACK_NAME \
18--capabilities      CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND
19echo "Done"

And then, execute the above script.

1chmod u+x ./deploy.sh
2./deploy.sh

After that, go to the IAM page in the AWS web site and generate a new key to access the above DynamoDB tables from some applications.

If you want to delete it, execute the following command.

1aws cloudformation delete-stack --stack-name "DynamoDBTableUserList"

Kubernetes: Suspend CronJobs and terminate current pod without restarting

Mon, 19 Oct 2020 22:09:00 +0900

It’s very easy to get information about how to suspend CronJob on Kubernetes clusters. It just executes the "kubectl patch" command. However, if its job is working at that time, the job will not be stopped and it’s impossible to terminate by "kubectl delete pod" command when the restartPolicy was set to "OnFailure" because even though the suspend flag modified to "true", a new pod will be created right after deleting. There’s no command or options to do that in kubectl command but here is the work around which I came across.

First I apply the following a test CronJob.

 1apiVersion: batch/v1beta1
 2kind: CronJob
 3metadata:
 4  name: cronjob-test
 5  namespace: app
 6spec:
 7  schedule: "15 * * * *"
 8  concurrencyPolicy: Forbid
 9  startingDeadlineSeconds: 60
10  successfulJobsHistoryLimit: 5
11  failedJobsHistoryLimit: 5
12  jobTemplate:
13    metadata:
14      name: cronjob-test
15    spec:
16      template:
17        spec:
18          restartPolicy: "OnFailure"
19          containers:
20          - name: cronjob-test
21            image: alpine:latest
22            imagePullPolicy: Always
23            command: ["tail", "-f", "/dev/null"]

And deploy it but I don’t want to modify the schedule manually and don’t want to wait long so I deploy it by the following script which sets the schedule to the next minute automatically.

 1#!/bin/bash
 2
 3# Get current hour and minute string
 4hour=`date '+%-H'`
 5min=`date '+%-M'`
 6
 7# Increment or reset to 0
 8if [ $min -eq 59 ];then
 9    let hour++
10    min=0
11else
12    let min++
13fi
14
15# Deploy
16cat cronjob.yml \
17| sed "s/\(schedule: \).*/\1${min} * * * */g" \
18| kubectl apply -f - || exit 1
19
20printf "Next schedule is %02d:%02d\n" $hour $min

And then, a new pod will be created and it will run continuously. After that, modify the restart_policy to "Never".

1kubectl get cronjobs.batch cronjob-test -o yaml \
2| sed 's/\(restartPolicy: \).*/\1Never/g' \
3| kubectl apply -f -

Then, modify the suspend flag.

1kubectl patch cronjobs $job_name -p '
2{
3    "spec":{
4        "suspend":'true'
5    }
6}'

However, the current working pod is still working. So, it should be terminated but the target is the job not the pod.

1# Get job batch ID which is generated by the CronJob
2job_batch_id=`kubectl get cronjobs.batch $job_name -o json | jq -r '.status.active[0].name'`
3
4# Delete the Job by the above ID
5kubectl delete "job.batch/${job_batch_id}"

After the above, the current working pod will be disappeared and not recreated automatically. Of course, the next scheduled time arrived, a new pod will not be created because the suspend flag is set to "true" now.

When roll back the settings, should to do like this.

 1## suspend flag
 2kubectl patch cronjobs $job_name -p '
 3{
 4    "spec":{
 5        "suspend":false
 6    }
 7}'
 8
 9## restartPolicy
10kubectl get cronjobs.batch cronjob-test -o yaml \
11| sed 's/\(restartPolicy: \).*/\1OnFailure/g' \
12| kubectl apply -f -

And then, when the next scheduled date comes, a new pod will be created.

Should to avoid to use public docker image on productions straightly

Sat, 17 Oct 2020 01:11:00 +0900

There are a lot of samples of Kubernetes and Docker Compose which define public images like nginx:alpine, golang php-fpm and so on. However even though it’s easy to use, it’s not good for production environments because there's a possibility that they are not up-to-date.

For example, I’m trying with one of the famous official docker images "nginx:alpine" which is a lightweight NGINX image. I ran the image and executed the "apk upgrade" command, there were 10 packages that should be updated.

 1$ docker run -it --rm nginx:alpine sh -c "apk upgrade --update --no-cache"
 2
 3fetch https://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
 4fetch https://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
 5(1/10) Upgrading musl (1.1.24-r8 -> 1.1.24-r9)
 6(2/10) Upgrading busybox (1.31.1-r16 -> 1.31.1-r19)
 7Executing busybox-1.31.1-r19.post-upgrade
 8(3/10) Upgrading alpine-baselayout (3.2.0-r6 -> 3.2.0-r7)
 9Executing alpine-baselayout-3.2.0-r7.pre-upgrade
10Executing alpine-baselayout-3.2.0-r7.post-upgrade
11(4/10) Upgrading ca-certificates-bundle (20191127-r2 -> 20191127-r4)
12(5/10) Upgrading ssl_client (1.31.1-r16 -> 1.31.1-r19)
13(6/10) Upgrading libcurl (7.69.1-r0 -> 7.69.1-r1)
14(7/10) Upgrading curl (7.69.1-r0 -> 7.69.1-r1)
15(8/10) Upgrading musl-utils (1.1.24-r8 -> 1.1.24-r9)
16(9/10) Upgrading brotli-libs (1.0.7-r5 -> 1.0.9-r1)
17(10/10) Upgrading libxml2 (2.9.10-r4 -> 2.9.10-r5)
18Executing busybox-1.31.1-r19.trigger
19Executing ca-certificates-20191127-r4.trigger
20OK: 24 MiB in 42 packages

So, even though it’s unnecessary to add any packages and copy files, I usually create a Dockerfile like this.

1FROM nginx:alpine
2RUN apk upgrade --update --no-cache

And then, build and register it to my private image registry and all pods use this.

On the other hand, I don’t use the "latest" tag because of the following reasons.

It will be updated unexpectedly
It’s hard to decide which version which I have to roll back when some troubles happen

There’s one more thing which I usually pay attention to is the variations of base distribution. For example, the default base image of the Golang official image is Debian and also there are tags which are based on Alpine Linux. When using the Alpine based images, should to pay attention to tags because there are tags which Alpine’s version is included and not included one as follows.

NOT Included：1.14.10-alpine, 1.14-alpine
Included：1.14.10-alpine3.11, 1.14-alpine3.11, 1.14.10-alpine3.12, 1.14-alpine3.12

In this case, should use the included one because if it’s not, one day the base image will be updated unexpectedly and it will be cause of build failure or execution failure of applications.

Show image tags for each revisions of Kubernetes deployments

Fri, 16 Oct 2020 16:11:00 +0900

One of the useful and powerful function is the roll back for me. Basically, all applications on our cluster are deployed automatically by CodePipeline in AWS but if some trouble happen, rolling back to the previous version is one of our options. At that time, all I have to do is the following command.

1$ kubectl rollout undo deployment my-app

When I execute the following command, the list of the revisions which are still remaining on the database on the Kubernetes cluster will be displayed.

1$ kubectl rollout history deployment my-app

The command’s response is:

 1REVISION  CHANGE-CAUSE
 229        <none>
 330        <none>
 431        <none>
 532        <none>
 633        <none>
 734        <none>
 835        <none>
 936        <none>
1037        <none>
1138        <none>
1239        <none>

And then, if the application should be rolled back to the specific revisions, the following command will work.

1$ kubectl rollout undo deployment my-app --to-revision=35

However, I want to know which image was used for each revision because we use the date string like "yyyymmddHHMMSS" for image tags as I posted before.

Docker: What’s better image tag than latest, none, and versions?

So, If I can see the list of revision numbers and image tags will be helpful for me. However, there’s no function in the kubectl command so I made a shell script like this.

 1#!/bin/bash
 2deployment_name=${1:-NONE}
 3
 4if [ $deployment_name == NONE ];then
 5    echo "USAGE: $0 DEPLOYMENT_NAME"
 6    exit 1
 7fi
 8
 9resource_name=`kubectl rollout history deployment $deployment_name -o name`
10echo "Resource name: $resource_name"
11for rev in `kubectl rollout history $resource_name | egrep '^[0-9]+' | awk '{print $1}'`
12do
13    image=`kubectl rollout history $resource_name --revision=${rev} | grep Image: | awk '{print $2}'`
14    printf "%02d : %s\n" $rev $image
15done

And its response is like this.

 1Resource name: deployment.apps/my-app
 232 : my-registry/my-app:202010061110
 333 : my-registry/my-app:202010070906
 434 : my-registry/my-app:202010080051
 535 : my-registry/my-app:202010080202
 637 : my-registry/my-app:202010130703
 738 : my-registry/my-app:202010130728
 839 : my-registry/my-app:202010080300
 940 : my-registry/my-app:202010130754
1041 : my-registry/my-app:202010130823
1142 : my-registry/my-app:202010130823
1243 : my-registry/my-app:202010130940

As a result of this, I can choose the relevant revision by the deployment date.

Run Memcached persistently on Kubernetes

Thu, 15 Oct 2020 16:45:00 +0900

When running Memcached on the Kubernetes cluster, the cache data will disappear whenever the pod restarts. I know that Redis has the built-in replication and it can be able to have data on persistence disks. However, I have some reasons to run the Memecached on our Kubernetes cluster. So, I was thinking about it and I came across my mind the way using postStart and preStop in lifecycle settings.

Before explaining, the disadvantages of this way are as follows.

Cannot cope with node failures
Need to add a persistent volume

First, create a script file that loads a dump file if it exists, and then output an empty file for the readiness probe. (post_start.sh)

 1#!/bin/ash
 2memc_host="127.0.0.1"
 3memc_port=11211
 4
 5# Load previous prestop data
 6dump_file="${DUMP_DIR}/${HOSTNAME}.dump.prestop"
 7if [ -f $dump_file ];then
 8    echo "Loading previous prestop data..."
 9    cat $dump_file | nc $memc_host $memc_port
10fi
11
12# For readiness probe
13touch /tmp/ready

And then, create a script that dump data before termination.

1#!/bin/ash
2
3# Load previous prestop data
4/memcached-tool ${HOSTNAME} dump > ${DUMP_DIR}/${HOSTNAME}.dump.prestop

After creating the above scripts, build the container image with the scripts by the following Dockerfile.

 1FROM memcached:1.6.6-alpine
 2
 3# Switch to root for installing packages
 4USER root
 5
 6RUN apk --update --no-cache add \
 7        bash \
 8        busybox-extras \
 9        perl
10
11COPY memcached-tool  /
12COPY post_start.sh   /
13COPY pre_stop.sh     /
14
15# Switch to memcache for running
16USER memcache

Finally, create the following YAML file and deploy it to the Kubernetes cluster.

 1apiVersion: apps/v1
 2kind: StatefulSet 
 3metadata:
 4  name: memcached
 5  namespace: app
 6  labels:
 7    app: memcached
 8spec:
 9  replicas: 2
10
11  selector:
12    matchLabels:
13      app: memcached
14
15  serviceName: memcached
16  template:
17    metadata:
18      labels:
19        app: memcached
20    spec:
21      affinity:
22        podAntiAffinity:
23          requiredDuringSchedulingIgnoredDuringExecution:
24          - labelSelector:
25              matchExpressions:
26              - key: app
27                operator: In
28                values:
29                - memcached
30            topologyKey: "kubernetes.io/hostname"
31
32      containers:
33        - name: memcached
34          image: registry/my-memcached:202010051112
35          imagePullPolicy: IfNotPresent
36          args: ["-U", "0", "-B", "ascii"]
37
38          ports:
39            - containerPort: 11211 
40
41          env:
42            - name: DUMP_DIR
43              value: "/var/tmp/memcache-dump"
44
45          resources:
46            requests:
47              cpu: 10m
48              memory: 16Mi
49
50          livenessProbe:
51            tcpSocket:
52              port: 11211
53            initialDelaySeconds: 2 
54            periodSeconds: 5
55
56          readinessProbe:
57            exec:
58              command:
59              - cat
60              - /tmp/ready
61            initialDelaySeconds: 5
62            periodSeconds: 5
63   
64          lifecycle:
65            postStart:
66              exec:
67                command: ["/post_start.sh"]
68            preStop:
69              exec:
70                command: ["/pre_stop.sh"]
71
72          volumeMounts:
73            - name: "memcache-dump"
74              mountPath: "/var/tmp/memcache-dump"
75
76      volumes:
77        - name: "memcache-dump"
78          persistentVolumeClaim:
79            claimName: memcache-efs

I know this is not a sophisticated way but it’s easy to create. So, I think it’s useful in some situations.

Kubernetes manifest to prevent service unavailable

Wed, 14 Oct 2020 16:55:00 +0900

Once build the CI/CD which runs automatically from build to deployment, developers are afraid that some problematic container image will be deployed and it will cause service unavailable. Especially web sites should be checked by some scraping tool like Selenium but it’s hard to do that for most start-up companies because of human resources. However, adding some settings to the Kubernetes manifest will reduce possibilities of problems.

First, configure the rolling update settings like this.

1  strategy:
2    type: RollingUpdate
3    rollingUpdate:
4      maxSurge: 1
5      maxUnavailable: 0

Second, configure the liveness probe as follows.

1          livenessProbe:
2            tcpSocket:
3              port: 80
4            initialDelaySeconds: 15
5            periodSeconds: 20

If the running application is simple, it’s enough until here but if it’s complex, should add some configuration. For example, cache server’s pods which have to do the pre cache loading process before service-in should add a readiness probe.

Create an empty dir to share the readiness probe file in advance.

1      volumes:
2        - name: "readiness-check"
3          emptyDir: {}

And then, add a sidecar to run the script which executes pre cache loading and then create an empty file for readiness check.

1    - name: init-cache
2      image: myregistry/webcache-init::202007301510
3      command: ["/bin/ash", "-c", "/init.sh && tail -f /dev/null"]
4
5      volumeMounts:
6        - mountPath: "/tmp/readiness-check"
7          name: "readiness-check"

And also, add the mount setting and the readiness check to the main app container setting.

 1        - name: webcache
 2          image: myregistry/webcache:202007301510
 3          imagePullPolicy: IfNotPresent
 4          ports:
 5            - containerPort: 80
 6          readinessProbe:
 7            exec:
 8              command:
 9              - cat
10              - /tmp/readiness-check/init_cache_ready
11            initialDelaySeconds: 900
12            periodSeconds: 5
13
14
15          volumeMounts:
16            - mountPath: "/tmp/readiness-check"
17              name: "readiness-check"

By the way, from version 1.16, there's a new feature "Startup Probe." It looks good to configure more flexibly. Actually I haven't tried yet because my production cluster is 1.14.

Ok, here is the full YAML file.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  name: webcache
 5  namespace: app
 6  labels:
 7    app: webcache
 8spec:
 9  replicas: 4
10  selector:
11    matchLabels:
12      app: webcache
13  strategy:
14    rollingUpdate:
15      maxSurge: 1
16      maxUnavailable: 0
17    type: RollingUpdate
18  template:
19    metadata:
20      labels:
21        app: webcache
22    spec:
23      affinity:
24        podAntiAffinity:
25          requiredDuringSchedulingIgnoredDuringExecution:
26          - labelSelector:
27              matchExpressions:
28              - key: app
29                operator: In
30                values:
31                - webcache
32            topologyKey: "kubernetes.io/hostname"
33
34      containers:
35        - name: webcache
36          image: myregistry/webcache:202007301510
37          imagePullPolicy: IfNotPresent
38          ports:
39            - containerPort: 80
40          readinessProbe:
41            exec:
42              command:
43              - cat
44              - /tmp/readiness-check/init_cache_ready
45            initialDelaySeconds: 900
46            periodSeconds: 5
47
48          resources:
49            requests:
50              cpu: 10m
51              memory: 64Mi
52            limits:
53              cpu: 1000m
54              memory: 1000Mi   
55
56          livenessProbe:
57            tcpSocket:
58              port: 80
59            initialDelaySeconds: 15
60            periodSeconds: 20
61
62          lifecycle:
63            preStop:
64              exec:
65                command: ["/bin/ash", "-c", "sleep 10"]
66          
67          volumeMounts:
68            - mountPath: "/tmp/readiness-check"
69              name: "readiness-check"
70
71        - name: init-cache
72          image: myregistry/webcache-init::202007301510
73          command: ["/bin/ash", "-c", "/init.sh && tail -f /dev/null"]
74
75          resources:
76            requests:
77              cpu: 10m
78              memory: 16Mi
79            limits:
80              cpu: 100m
81              memory: 32Mi   
82
83          volumeMounts:
84            - mountPath: "/tmp/readiness-check"
85              name: "readiness-check"
86
87      volumes:
88        - name: "readiness-check"
89          emptyDir: {}

Eliminate unnecessary instances from EC2 Service Discovery in Prometheus

Tue, 13 Oct 2020 00:00:00 +0900

EC2 Service Discovery is very useful in the Prometheus configuration especially when the autoscale is used. However, some temporary or test servers which do not need to be monitored will be cause of unnecessary alerts. So, I added some settings to avoid it.

For example, if I would like to eliminate instances which have names including "test" or "dev", I’ll add the following configurations.

1    relabel_configs:
2    - source_labels: [__meta_ec2_tag_Name]
3      regex: .*test.*
4      action: drop
5    - source_labels: [__meta_ec2_tag_Name]
6      regex: .*dev.*
7      action: drop

As a result of the above, the Prometheus will ignore those nodes. So, if I would like to use some instances which are temporary or no need to be monitored, the names should include one of them like "test-app" or "dev-web-server"

Fluentd: Variables in config are useful

Mon, 12 Oct 2020 00:07:00 +0900

Fluentd is necessary software for operating Kubernetes clusters. Variables in the Fluentd’s configs are useful because they reduce redundant writing and also it will come in handy to switch environments between development and production.

For example, I configure the fluentd which is running on our Kubernetes cluster as a DaemonSet as follows.

 1<match **>
 2  @type elasticsearch
 3  type_name _doc
 4  include_tag_key true
 5  host "#{ENV['ELASTICSEARCH_HOST']}"
 6  port "#{ENV['ELASTICSEARCH_PORT']}"
 7  time_key time
 8  reconnect_on_error true
 9  reload_on_failure true
10
11  ## Logstash format settings (index will be "${tag}-%Y.%m")
12  logstash_format true
13  logstash_prefix ${tag}
14  logstash_prefix_separator -
15  logstash_dateformat %Y.%m 
16  
17  <buffer tag, time> 
18    @type file
19    timekey 1h
20    path /var/log/fluentd-buffers/${tag}.buffer
21    flush_mode interval
22    retry_type exponential_backoff
23    flush_thread_count 2
24    flush_interval 5s
25    retry_forever
26    retry_max_interval 30
27    chunk_limit_size 32M
28    queue_limit_length 100
29    overflow_action drop_oldest_chunk
30  </buffer>
31</match>

As you see on line 5 and 6, I defined the ElasticSearch host and port by variables. And then, I configured the Kubernetes YAML file like this.

1        env:
2        - name: ELASTICSEARCH_HOST
3          value: "es.mydomain"
4        - name: ELASTICSEARCH_PORT
5          value: "9200"

On the other hand, the YAML file for the development environment is like this.

1　    env:
2        - name: ELASTICSEARCH_HOST
3          value: "host.docker.internal"
4        - name: ELASTICSEARCH_PORT
5          value: "9200"

So, this will be able to switch these environments easily.

Fluentd: Pitfalls of custom log format

Sun, 11 Oct 2020 00:00:00 +0900

One day, I realized that the fluentd on our Kubernetes cluster was logging a lot of "invalid date" errors like this.

1#0 dump an error event: error_class=ArgumentError error="invalid date" location="/usr/local/bundle/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:519:in `parse'" tag="Fluent::ElasticsearchOutput::TimeParser.error" time=2020-07-21 14:59:59.894536631 +0000 record={"tag"=>"hoge.fuga.kubernetes", "time"=>2020-07-21 14:59:52.000000000 +0000, "format"=>nil, "value"=>"1595343592"}

When ingesting HAProxy’s logs which I defined as JSON format, the error caused. I couldn’t understand why it happened because the above log said “TimeParser.error time=2020-07-21 14:59:59.894536631”. It looks the correct format that I expected. And also, other application logs like NGINX were no problem. Just this pod which has the HAProxy only.

I checked the log format again in the HAProxy configuration file and it was the follows.

1    log global
2    log-format '{"host":"%H","ident":"haproxy","pid":%pid,"time":"%Ts","haproxy":{"conn":{"act":%ac,"fe":%fc,"be":%bc,"srv":%sc},"queue":{"backend":%bq,"srv":%sq},"time":{"tq":%Tq,"tw":%Tw,"tc":%Tc,"tr":%Tr,"tt":%Tt},"termination_state":"%tsc","retries":%rc,"network":{"client_ip":"%ci","client_port":%cp,"frontend_ip":"%fi","frontend_port":%fp},"ssl":{"version":"%sslv","ciphers":"%sslc"},"request":{"method":"%HM","uri":"%[capture.req.uri,json(utf8s)]","protocol":"%HV","header":{"host":"%[capture.req.hdr(0),json(utf8s)]","xforwardfor":"%[capture.req.hdr(1),json(utf8s)]","referer":"%[capture.req.hdr(2),json(utf8s)]","user-agent":"%[capture.req.hdr(3),json(utf8s)]","content-length":"%[capture.req.hdr(4),json(utf8s)]"}},"name":{"backend":"%b","frontend":"%ft","server":"%s"},"response":{"status_code":%ST,"header":{"server":"%[capture.res.hdr(0),json(utf8s)]","content-type":"%[capture.res.hdr(1),json(utf8s)]","transfer-encoding":"%[capture.res.hdr(2),json(utf8s)]","cache-control":"%[capture.res.hdr(3),json(utf8s)]","via":"%[capture.res.hdr(4),json(utf8s)]","location":"%[capture.res.hdr(5),json(utf8s)]","content-encoding":"%[capture.res.hdr(6),json(utf8s)]"}},"bytes":{"uploaded":%U,"read":%B}}}'

After a few minutes, I realized the value of the “time” column was quoted by double quotations. So, I deleted it and deployed the HAProxy again then the errors disappeared. Finally I found out that the time column in the log format must not be quoted because that will be ingested as String not Date.

Kubernetes: How many pods are available per node on AWS EKS?

Sat, 10 Oct 2020 16:51:00 +0900

When getting started to use AWS EKS for building Kubernetes clusters, we should consider the limitations of the number of IP addresses it depends on the instance size. Actually, I didn’t know about it because I just overlooked it on the official document. I know it was my mistake but it should be a blind spot for some people so I’ll share my experience.

After a few months since I got started to use our cluster, I realize that there’s some error logs of kubelet like this.

1RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "xxxxxx" network for pod "hoge-1602061920-snvck": NetworkPlugin cni failed to set up pod "hoge-1602061920-snvck_apps" network: add cmd: failed to assign an IP address to container

I investigated this problem and then I found out there’s some limitation of the pod’s number by EC2 instance’s restrictions. EC2 instances has the limitation of the number of ENI (Elastic Network Interface) and how many IP addresses it’s able to have. The limitation is defined depending on the instance size. For example, m5.Large instance is able to have 3 ENIs and 10 IP addresses for each ENI so totally 30 addresses. However the primary address of each ENI cannot be used for Pods so 27 addresses will be available for pods. Another example, m5.4xlarge instance is able to have 8 ENIs and 30 IP addresses for each ENI so 232 addresses will be available for pods.

I’m not sure it’s enough compared to the resources like CPUs and memories for most people. However it should be considered before using EKS especially the cluster size is not large yet. In my case, after reaching the limitation, I’m operating as follows.

Use NodeAffinity option to avoid running multiple pods on the same node and don’t increase the replica number over the number of nodes
When running batches as CronJobs, adjust the schedule to avoid overlapping

References

AWS > Documentation > Amazon EC2 > User Guide for Linux Instances > Elastic network interfaces

HAProxy running on Kubernetes and output a lot of BADREQ logs

Fri, 09 Oct 2020 23:35:00 +0900

When I was setting up HAProxy running on our Kubernetes cluster, I defined the log format as JSON and I got a lot of BADREQ logs with status code 400 as follows.

 1{
 2  "conn": {
 3    "act": 7,
 4    "fe": 7,
 5    "be": 0,
 6    "srv": 0
 7  },
 8  "queue": {
 9    "backend": 0,
10    "srv": 0
11  },
12  "time": {
13    "tq": -1,
14    "tw": -1,
15    "tc": -1,
16    "tr": -1,
17    "tt": 0
18  },
19  "termination_state": "CR--",
20  "retries": 0,
21  "network": {
22    "client_ip": "172.31.79.179",
23    "client_port": 63740,
24    "frontend_ip": "172.31.76.184",
25    "frontend_port": 80
26  },
27  "ssl": {
28    "version": "-",
29    "ciphers": "-"
30  },
31  "request": {
32    "method": "<BADREQ>",
33    "uri": "-",
34    "protocol": "<BADREQ>",
35    "header": {
36      "host": "-",
37      "xforwardfor": "-",
38      "referer": "-"
39    }
40  },
41  "name": {
42    "backend": "https-in",
43    "frontend": "https-in",
44    "server": "<NOSRV>"
45  },
46  "response": {
47    "status_code": 400,
48    "header": {
49      "xrequestid": "-"
50    }
51  },
52  "bytes": {
53    "uploaded": 0,
54    "read": 245
55  }
56}

As a result of researching these error logs, I realize that the cause of these errors is the liveness probe of Kubernetes like this.

1        livenessProbe:
2          tcpSocket:
3            port: 80
4          initialDelaySeconds: 15
5          periodSeconds: 5

I’m not sure but I think health-check by TCP not HTTP will raise the same logs.

Finally, I solved this problem by adding "dontlognull" option like this.

1defaults
2    mode http
3    option dontlognull
4       :
5       :

Test web page on local K8s via HTTPS w/o specifying port

Tue, 06 Oct 2020 12:22:00 +0900

When I deploy some web application on the local Kubernetes cluster, I would like to test it like

Without specifying a port number
Via HTTPS
Specify original FQDN not localhost

For example, if I'm developing "www.example.com," I would like to test by the URL like https://www.example.work/. So, Let me share how to do it.

When deploying a web site on the Kubernetes on the local computer, I need to test the site with the URL like https://localhost:30000. But this is bothersome for testing because it depends on the web site, some links in the page don’t have the port number. And also, nowadays, most of the web site should be accessed via HTTPS, so I want to test the site via HTTPS. So I made a load balancer to solve the problem with NGINX.

At first, I put the settings for redirecting from HTTP to HTTPS with the 307 status like this.

 1server {
 2    listen       80;
 3    server_name  _;
 4
 5    location = /any.example.com.pem {
 6        root   /usr/share/nginx/html;
 7    }
 8
 9    location / {
10        return 307 https://$host$request_uri;
11    }
12}

The above setting includes a setting to download the dummy SSL cert file which I will explain later. And I set the redirecting status to 307, not 301 to keep the request's method and body just in case.

And then, I put the settings for HTTPS like this.

 1map $http_host $backend {
 2    default $http_host;
 3    example.work     web_example;
 4    example.com      web_example;
 5}
 6
 7upstream web_example {
 8    server host.docker.internal:30000;
 9}
10
11server {
12    listen       443 default ssl;
13    server_name  _;
14
15    # SSL
16    ssl_certificate      /usr/local/nginx/cert/any.example.com.pem;
17    ssl_certificate_key  /usr/local/nginx/cert/any.example.com.key;  
18
19    location / {
20        proxy_set_header Host                   $host;
21        proxy_set_header X-Real-IP              $remote_addr;
22        proxy_set_header X-Forwarded-Proto      https;
23        proxy_set_header X-Forwarded-Host       $host;
24        proxy_set_header X-Forwarded-Server     $host;
25        proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
26
27        proxy_pass https://$backend;
28
29        send_timeout          180;
30        proxy_connect_timeout 600;
31        proxy_read_timeout    600;
32        proxy_send_timeout    600;
33    }
34}

The above setting accepts the following FQDNs.

1www.example.com
2www.example.work

because when developing on the local environment, some slightly different FQDN, www.example.work in this case, is very useful for avoiding to modify the "hosts file" time and again. After accepting the request, NGINX transfers it to 30000 port on the localhost with some additional headers. The port should be changed depending on your target.

Before running the above NGINX, I need to create a self-cert file. I'll create it with a simple docker image like this.

1FROM alpine:latest
2
3RUN apk update \
4    && apk add --no-cache \
5                openssl

And then, create an OpenSSL setting file like this.

 1[req]
 2default_bits       = 2048
 3distinguished_name = req_distinguished_name
 4req_extensions     = v3_req
 5x509_extensions    = v3_ca
 6prompt             = no
 7
 8[ v3_ca ]
 9subjectKeyIdentifier    = hash
10authorityKeyIdentifier  = keyid:always,issuer
11basicConstraints        = CA:FALSE
12
13[ req_distinguished_name ]
14countryName             = JP
15stateOrProvinceName     = Tokyo
16localityName            = Shibuya
17organizationName        = Example Dot Com
18organizationalUnitName	= Infrastructure 
19commonName              = example.com
20
21[ v3_req ]
22# Extensions to add to a certificate request
23basicConstraints    = CA:FALSE
24keyUsage            = nonRepudiation, digitalSignature, keyEncipherment
25extendedKeyUsage    = clientAuth, serverAuth
26subjectAltName      = @alt_names
27
28[ alt_names ]
29DNS.1 = localhost
30DNS.2 = example.com
31DNS.3 = *.example.com
32DNS.4 = example.work
33DNS.5 = *.example.work

And then, create docker-entrypoint.sh to generate cert files with the above setting file.

 1#!/bin/ash
 2dir="/cert"
 3prefix="any.example.com"
 4file_path_prefix="${dir}/${prefix}"
 5
 6# Output files
 7intermediate_file="${dir}/dummy.cer"
 8cert_file="${file_path_prefix}.crt"
 9csr_file="${file_path_prefix}.csr"
10key_file="${file_path_prefix}.key"
11output_pem="${file_path_prefix}.pem"
12
13if [ ! -f $output_pem ];then
14    # Generate key and certificate files
15    openssl req -sha256 -new -newkey rsa:4096 -days 3650 -nodes -x509 \
16        -extensions v3_req \
17        -config /openssl.cnf \
18        -keyout $key_file \
19        -out    $cert_file
20
21    # Generate dummy files
22    touch $csr_file
23    touch $intermediate_file
24
25    # Gen integrated PEM file
26    cat \
27    $cert_file \
28    $intermediate_file \
29    $key_file \
30    > $output_pem
31else
32    echo "Already existed."
33fi
34
35# Keep running
36tail -f /dev/null

And then, create a docker-compose.yml to run the above 2 applications. Notice that it will use 80 and 443 ports so it’s necessary to quit in advance if some other application is using.

 1version: '3.8'
 2services:
 3  test-lb-nginx:
 4    image: nginx:alpine
 5    restart: always
 6    container_name: dev-elb
 7    ports:
 8      - 80:80
 9      - 443:443
10    volumes:
11      - ./docker/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
12      - ./docker/nginx/conf.d:/etc/nginx/conf.d:rw
13      - ./docker/nginx/cert:/usr/local/nginx/cert:ro
14      - ./docker/nginx/cert:/usr/share/nginx/html:ro
15    depends_on: 
16      - gen-self-cert
17
18  gen-self-cert:
19    build:
20      context: ./docker/self_cert
21    restart: always
22    container_name: gen-cert
23    volumes:
24      - ./docker/self_cert/docker-entrypoint.sh:/docker-entrypoint.sh:ro
25      - ./docker/self_cert/openssl.cnf:/openssl.cnf:ro
26      - ./docker/nginx/cert:/cert
27    entrypoint: "/docker-entrypoint.sh"

And run with the following command.

1$ docker-compose up -d

Access https://127.0.0.1/any.example.com.pem to get the generated cert file and then it should be imported to the system. (The way to import is different it depends on your OS)

Before accessing the above containers, it's necessary to add some FQDNs which I want to test like this.

1127.0.0.1 www.example.work

Finally, it's possible to get access by the following URL without specifying port via HTTPS. https://www.example.work/

Docker: What’s better image tag than latest, none, and versions?

Sun, 04 Oct 2020 01:11:00 +0900

In my opinion, and based on the experiences that I've operated our Kubernetes cluster on AWS EKS for my company's service, docker image tag should be date string like "yyyymmddHHMMSS". So, let me share why I'm thinking about it.

I know that most of the docker images of open source software like NGINX, MySQL, Go, PHP are version numbers or combined with base image names like Debian, Alpine. That’s why at first I put version number like "appname:1.0" but I realize that it’s not good because of the following reasons

Hard to understand when the currently running image was built
Need to concern about versioning rule when building the image
When using CI/CD pipeline, need to define the version numbering rule in advance
If somebody put a version manually, there's a possibility to make a mistake

So, I came up with using a date string as a tag. For example, if the current date and time is 20/Sep/2020 12:30 I put the "test-app:202009201230" tag to the image. But it's bothering to put it manually, so I make a script like this.

 1#!/bin/bash
 2export DOCKER_BUILDKIT=1
 3ECR_FQDN='000000000000.dkr.ecr.us-west-2.amazonaws.com'
 4VER_DATE=`date '+%Y%m%d%H%M'`
 5APP='test-app'
 6TAG="${ECR_FQDN}/${APP}:${VER_DATE}"
 7
 8docker build -t $TAG . \
 9&& docker push $TAG \
10&& sed -i "s;\(image: \).*;\1${TAG};g" ./deployment_prod.yml

Of course, it depends on the projects, if its release version is managed very strictly, the version number will be good for the tag. But the project is updated and released frequently, I think the date string is better. Also, I usually avoid using the "latest" tag because it's hard to grasp which version is running now. The other problem of the "latest" tag is when I deploy on the Kubernetes cluster, it will not be a trigger to renew pods.

tech-tips on takac.dev

Docker: How to solve the trouble of SSH tunnel connection?

Docker: How to install packages to busybox image?

Prometheus: How to solve blackbox exporter icmp (ping) failure

Fluentd: Warning on parser filter: nested repeat operator '+' and '*' was replaced with '*' in regular expression

PHP: Fatal error when build docker image with composer

Python: Got AttributeError when accessing App Store Connect API using appstoreconnect library

Example of Selenium with Python on Docker with latest FireFox

Reference

Ruby: AWS SAM build error with mongoid package

Ruby: bundle install hangs indefinitely with high CPU usage

Docker: What should I do when getting "503 service unavailable" error after executing docker pull command?

Docker: How to solve Apache 403 Forbidden error?

Docker: Run apache as non-root user based on the official image

AWS: Create external Aurora replication host by CLI

Docker Hub: Error 429 too many requests

Kibana query language cheat sheet for web service

Hide empty line

Redirected HTTP access

Must and Must Not

Exclude bot logs by UA

AWS: CloudFormation example creating DynamoDB table and IAM

Kubernetes: Suspend CronJobs and terminate current pod without restarting

Should to avoid to use public docker image on productions straightly

Show image tags for each revisions of Kubernetes deployments

Run Memcached persistently on Kubernetes

Kubernetes manifest to prevent service unavailable

Eliminate unnecessary instances from EC2 Service Discovery in Prometheus

Fluentd: Variables in config are useful

Fluentd: Pitfalls of custom log format

Kubernetes: How many pods are available per node on AWS EKS?

HAProxy running on Kubernetes and output a lot of BADREQ logs

Test web page on local K8s via HTTPS w/o specifying port

Docker: What’s better image tag than latest, none, and versions?

Fluentd: Warning on parser filter: nested repeat operator '+' and '' was replaced with '' in regular expression