trouble-shooting on takac.dev

Docker: How to solve the trouble of SSH tunnel connection?

Thu, 16 Dec 2021 23:00:00 +0900

I wanted to connect to some external site via my server in AWS because the site has IP address filters as security. However my apartment doesn't have a static IP address so I came up with an idea connecting to the site via SSH tunnel and if it runs as a container in docker-compose, it’s very easy to execute by the command “docker-compose up -d.” But there was a pitfall when digging an SSH tunnel on docker. Let me share what the trap was and how to solve it.

The command which I executed on the CLI was like this.

1ssh -4f -NL 20000:hogeadmin.work:443 takashi@myadmin.server -p 10023

So, I made a simple container with the following Dockerfile to execute ssh client.
[ssh-tunnel/Dockerfile]

 1FROM alpine:latest
 2
 3RUN set -x \
 4    && apk update \
 5    && apk upgrade \
 6    && apk add --no-cache \
 7            openssh-client \
 8            ca-certificates \
 9            bash \
10            bind-tools
11
12EXPOSE 20000-20010
13
14ENTRYPOINT [ "/docker-entrypoint.sh" ]

And, I made a simple script as docker-entrypoint.sh.
[ssh-tunnel/docker-entrypoint.sh]

 1#!/bin/bash
 2REMOTE_HOST='x.x.x.x'
 3REMOTE_USER='user-name'
 4REMOTE_PORT='10023'
 5
 6connect_ssh_tunnel(){
 7    local local_port=$1; shift
 8    local remote_port=$1; shift
 9    local fqdn=$1; shift
10
11    ssh -4f -p ${REMOTE_PORT} -NL ${local_port}:${fqdn}:${remote_port} ${REMOTE_USER}@${REMOTE_HOST}
12}
13
14connect_ssh_tunnel '20000' '443' hogeadmin.work'
15
16# To keep running
17tail -f /dev/null

And then, I made a docker-compose.yml file like this.

 1version: '3.4'
 2services:
 3  ssh-tunnel:
 4    build:
 5      context: ./docker/ssh-tunnel
 6    restart: always
 7    container_name: ssh-tunnel
 8    ports:
 9      - 443:20000
10    volumes:
11      - ./docker/ssh-tunnel/docker-entrypoint.sh:/docker-entrypoint.sh:ro
12      - ~/.ssh:/root/.ssh:ro

So, I run the command “docker-compose up -d” to run the container. And then, I tried to connect via curl command with the “--resolve” option.

1$ curl --resolve hogeadmin.work:443:127.0.0.1 https://hogeadmin.work/
2curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to hogeadmin.work

I was confused because it’s just a simple thing but I got the above error. So, I tried to connect via the telnet command to check the port is alive or not.

1$ telnet localhost 443
2Trying ::1...
3Connected to localhost.
4Escape character is '^]'.
5Connection closed by foreign host.

The connection was closed quickly. So, I tried to do the same thing in the container.

 1$ docker exec -it ssh-tunnel bash
 2
 3bash-5.1# apk update && apk add -f curl
 4fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
 5fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
 6v3.14.3-58-g7fc21b9dfb [https://dl-cdn.alpinelinux.org/alpine/v3.14/main]
 7v3.14.3-57-g005638434d [https://dl-cdn.alpinelinux.org/alpine/v3.14/community]
 8OK: 14942 distinct packages available
 9OK: 21 MiB in 40 packages
10
11bash-5.1# curl localhost:20000
12<html>
13<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
14<body>
15<center><h1>400 Bad Request</h1></center>
16<center>The plain HTTP request was sent to HTTPS port</center>
17</body>
18</html>

I got the HTTP response as the above. So, the SSH tunnel in itself works correctly but it’s not able to connect via the mapped port which is defined in the docker-compose.yml file. So, I searched about the ssh client command and I realized that the “-L” option has the optional parameter called “bind_address.” So, I add "0.0.0.0" as bind address to the command in the script like this.

1connect_ssh_tunnel(){
2   local local_port=$1; shift
3   local remote_port=$1; shift
4   local fqdn=$1; shift
5 
6   ssh -4f -p ${REMOTE_PORT} -NL 0.0.0.0:${local_port}:${fqdn}:${remote_port} ${REMOTE_USER}@${REMOTE_HOST}
7}

And then, I restarted the docker-compose and tried again.

 1$ curl --resolve hogeadmin.work:443:127.0.0.1 https://hogeadmin.work/
 2<!DOCTYPE HTML>
 3<html lang="ja">
 4    <head>
 5        <meta charset="utf-8">
 6        </meta>
 7    </head>
 8    <body>
 9        Forbidden
10    </body>
11</html>

Finally, I got the HTTP response so it works as I expected. However this way is simple way so it has some problems as follows.

It’s impossible to get access to some multiple sites at the same time
It will be a problem when the 443/tcp port is already used in other purpose

But I have a solution that is a little bit complicated so I’ll share the way in another article in this blog in the near future.

Prometheus: How to solve blackbox exporter icmp (ping) failure

Wed, 28 Jul 2021 23:00:00 +0900

I set up my blackbox exporter to monitor some network appliances via ICMP but its RTT was always zero. I'll share how to investigate this problem and solve it.

First I added the following module setting to blackbox.yml because I've never used ICMP monitoring before.

1modules:
2    icmp:
3        prober: icmp
4        timeout: 5s

Then, I added the following setting to the Prometheus configuration file.

1  - job_name: 'icmp_check'
2    metrics_path: /probe
3    params:
4      module: [icmp]
5    static_configs:
6      - targets:
7        - 'target01.hoge'
8        - 'target02.hoge'

I restarted the Prometheus and the blackbox exporter and Prometheus got started to monitor the above two network appliances. Then, I searched its result by the following query on Prometheus's web UI.

1probe_icmp_duration_seconds{job="icmp_check", phase="rtt"}

However, the second one's value had not been captured correctly even though the first one was correct. Actually the 'rtt' was always zero. So, I tried to execute the ping command in the blackbox-exporter's container but the result was no problem. I was not sure why it happened so I decided to add the debug flag to the blackbox-exporter in my docker-compose.yml file like this.

 1 blackbox-exporter:
 2  container_name: blackbox-exporter 
 3  build: ./blackbox_exporter
 4  restart: always
 5  ports:
 6   - "9115:9115"
 7  command:
 8   - "--log.level=debug"
 9  sysctls:
10    net.ipv4.ping_group_range: "0 2147483647"

After restarting the blackbox-exporter, I got the following logs.

1ts=2021-07-01T03:00:00.890Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Beginning probe" probe=icmp timeout_seconds=5
2ts=2021-07-01T03:00:00.890Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolving target address" ip_protocol=ip6
3ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolved target address" ip=xxxx:xxxx:xxxx:xxxx::1
4ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating socket"
5ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating ICMP packet" seq=57536 id=40265
6ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Writing out packet"
7ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Error writing to socket" err="write udp [::]:560->[xxxx:xxxx:xxxx:xxxx::1]:0: sendto: cannot assign requested address"
8ts=2021-07-01T03:00:00.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Probe failed" duration_seconds=0.001720556
9ts=2021-07-01T03:07:19.099Z

And I realized that the second one's FQDN was solved as an IPv6 address. It caused the error because Docker uses the IPv4 only as default. So, it couldn't create a socket to send ICMP packets. That's why I added the following setting to the blackbox.yml file to use IPv4 preferably.

1modules:
2    icmp:
3        prober: icmp
4        timeout: 5s
5        icmp:
6            preferred_ip_protocol: "ip4"

Then, I restarted the blackbox-exporter again and checked the logs.

1ts=2021-07-01T03:16:27.891Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Beginning probe" probe=icmp timeout_seconds=5
2ts=2021-07-01T03:16:27.891Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolving target address" ip_protocol=ip4
3ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Resolved target address" ip=xxx.xxx.xxx.xxx
4ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating socket"
5ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Creating ICMP packet" seq=39633 id=48931
6ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Writing out packet"
7ts=2021-07-01T03:16:27.892Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Waiting for reply packets"
8ts=2021-07-01T03:16:27.997Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Found matching reply packet"
9ts=2021-07-01T03:16:27.997Z caller=main.go:180 module=icmp target=target02.hoge level=debug msg="Probe succeeded" duration_seconds=0.106222366

The exporter used the IPv4 address as I expected. So, I searched its result by the following query on the web UI again.

1probe_icmp_duration_seconds{job="icmp_check", phase="rtt"}

Finally, the value has been captured correctly. Last, don't forget to turn off the debug log setting for blackbox-exporter to avoid filling up the disk.

1 blackbox-exporter:
2  container_name: blackbox-exporter 
3  build: ./blackbox_exporter
4  restart: always
5  ports:
6   - "9115:9115"
7  # command:
8  #  - "--log.level=debug"

Fluentd: Warning on parser filter: nested repeat operator '+' and '' was replaced with '' in regular expression

Wed, 19 May 2021 00:35:00 +0900

I needed to add parser settings for CoreDNS working on my Kubernetes cluster for troubleshooting but I got an error "nested repeat operator '+' and '' was replaced with '' in regular expression." I had no idea why it happened because the additional regular expression was valid. The cause of the error was my mistake so let me share what’s the cause of?

First, I added log setting to the CoreDNS configuration like this.

 1$ kubectl get configmaps -n kube-system coredns -o yaml
 2apiVersion: v1
 3data:
 4  Corefile: |
 5    .:53 {
 6        log
 7
 8        errors
 9        health {
10           lameduck 5s
11        }
12        ready
13     :
14     :

And then, I restarted the pod and got logs like this.

1[INFO] 172.31.57.199:54169 - 1916 "A IN dynamodb.us-west-2.amazonaws.com. udp 50 false 512" NOERROR qr,rd,ra 98 0.000921506s

So, I create a regular expression string and test it on https://fluentular.herokuapp.com/. Then, added the following filter to the Fluentd configuration.

 1<filter **>
 2    @type parser
 3    key_name log
 4    reserve_data true
 5    replace_invalid_sequence true
 6    <parse>
 7        @type multi_format
 8        <pattern>
 9            format regexp
10            expression /^\[(?<severity>\S+)\] (?<remote>\d+\.\d+\.\d+\.\d+):(?<port>\d+) +- (?<id>\d+) +*"(?<type>\S+) +(?<class>\S+) +(?<name>[^ ]+)\. +(?<proto>\S+) +(?<size>\d+) +(?<do>\S+) +(?<bufsize>\d+)" +(?<rcode>\S+) +(?<rflags>[^ ]+) +(?<rsize>\d+) +(?<duration>\d+\.\d+)s$/
11        </pattern>
12     :
13     :

After reloading the fluentd pod, then the following error appeared.

12021-05-17 08:57:06 +0000 [info]: adding filter pattern="**" type="parser"
2/usr/local/bundle/gems/fluentd-1.11.4/lib/fluent/config/types.rb:92: warning: nested repeat operator '+' and '*' was replaced with '*' in regular expression

So, I deleted the expression and then I paste each element of the regular expression from the beginning one by one and then I found out the cause of the problem is "(?\d+) +*"(?\S+)." As you see, there is "+*" as my mistake. So, I modify the part like this.

1^\[(?<severity>\S+)\] (?<remote>\d+\.\d+\.\d+\.\d+):(?<port>\d+) +- (?<id>\d+) +"(?<type>\S+) +(?<class>\S+) +(?<name>[^ ]+)\. +(?<proto>\S+) +(?<size>\d+) +(?<do>\S+) +(?<bufsize>\d+)" +(?<rcode>\S+) +(?<rflags>[^ ]+) +(?<rsize>\d+) +(?<duration>\d+\.\d+)s$

And then, I restarted the pod and the errors didn’t appear.

PHP: Fatal error when build docker image with composer

Fri, 07 May 2021 00:25:00 +0900

I saw this news about the serious vulnerability in composer which is PHP package management software.

https://www.bankinfosecurity.com/php-composer-flaw-that-could-affect-millions-sites-patched-a-16523

So, I wanted to update composer in my Dockerfile but I realized that I already specified the version as follows.

1curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/bin --filename=composer --version=1.10.16

Because it was a workaround for an error related to hirak/prestissimo which I did before. So, I deleted it and built the Docker image again but I got a different error as follows.

1#14 281.5  122/122 [============================] 100%  - Installing kylekatarnls/update-helper (1.2.0): Extracting archive
2#14 281.5 
3#14 281.5 Fatal error: Class UpdateHelper\ComposerPlugin contains 2 abstract methods and must therefore be declared abstract or implement the remaining methods (Composer\Plugin\PluginInterface::deactivate, Composer\Plugin\PluginInterface::uninstall) in …

It looks some problem about kylekatarnls/update-helper library. So, I searched about the library and I found a clue on GitHub.

https://github.com/kylekatarnls/update-helper/issues/7

So, I deleted composer.lock file and built the image again. Then the problem was solved. The above page said I need to delete the ‘vendor’ directory but it’s not necessary for docker build because it will be generated newly.

Python: Got AttributeError when accessing App Store Connect API using appstoreconnect library

Thu, 06 May 2021 23:55:00 +0900

I created a small Python program to get data via the App Store Connect API for report automation with the appstoreconnect library. After update recently, I got the following error.

 1Traceback (most recent call last):
 2  File "/app/./main.py", line 88, in <module>
 3    APP_STORECONNECT_API.execute()
 4  File "/app/app_store_connect.py", line 176, in execute
 5    api = Api(self.API_KEY, self.CERT_FILE, self.ISSUER_ID)
 6  File "/usr/local/lib/python3.9/site-packages/appstoreconnect/api.py", line 51, in __init__
 7    token = self.token  # generate first token
 8  File "/usr/local/lib/python3.9/site-packages/appstoreconnect/api.py", line 269, in token
 9    self._token = self._generate_token()
10  File "/usr/local/lib/python3.9/site-packages/appstoreconnect/api.py", line 64, in _generate_token
11    return jwt.encode({'iss': self.issuer_id, 'exp': exp, 'aud': 'appstoreconnect-v1'}, key,
12AttributeError: 'str' object has no attribute 'decode'

It depends on the above error, the problem happened here.

https://github.com/Ponytech/appstoreconnectapi/blob/0916ac75790b13cf6c89617209a1a6c2db1f16d1/appstoreconnect/api.py#L79-L80

And the author already realized about this issue.

https://github.com/Ponytech/appstoreconnectapi/issues/37

He said the cause of the problem is the type of return value was changed in the JWT library from byte to string since 2.0. So, I thought I should specify a version of JWT which is the latest before 2.0. And then, I checked the version history here.

https://pypi.org/project/PyJWT/#history

The latest 1.x version is 1.7.1 so I specified it explicitly when executing the pip command in my Dockerfile like this.

1pip install PyJWT==1.7.1

The problem was solved. So, this is one of the workarounds for now about this issue.

Ruby: AWS SAM build error with mongoid package

Fri, 05 Feb 2021 23:59:00 +0900

Same as my last post , I’m not a Ruby developer but I had an opportunity to help move some Ruby applications from EC2 to Lambda on AWS. At the time, I had a problem when I built a Ruby application with AWS SAM framework. Let me share why it happened and how I fixed it.

I executed the build command "sam build‘ for the Ruby application but it failed with the following error message.

1Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "RubyBundlerBuilder:CopySource - [Errno 2] No such file or directory: '/tmp/samcli/source/vendor/bundle/ruby/2.7.0/gems/mongo-2.14.0/spec/support/ocsp'"}}
2
3Build Failed
4Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 72694, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1, 'requestId': 'c9a32492-242a-460b-8633-9c2b8c4b7650', 'installationId': 'a15de333-32e0-4651-b78a-c0843ab41c31', 'sessionId': '970cf38b-8715-42bb-b726-c664597c56ab', 'executionEnvironment': 'CLI', 'pyversion': '3.8.7', 'samcliVersion': '1.15.0'}}]}

It was weird for me because all package files were already installed via "bundle install" command. However, the above error said "No such file or directory" and the file is the following.

1'/tmp/samcli/source/vendor/bundle/ruby/2.7.0/gems/mongo-2.14.0/spec/support/ocsp'

In my Gemfile, I specified 'mongoid' and when I executed the "bundle install" command, the ‘mongo’ package was also installed because of the dependency. The automatically installed version was "2.14.0" which is the latest version and I checked the above file path in its GitHub repository.

https://github.com/mongodb/mongo-ruby-driver/blob/master/spec/support/ocsp

And then, I realized that the file is a symbolic link to the following path.

1../../.mod/drivers-evergreen-tools/.evergreen/ocsp

And the .mod directory in the above path and "drivers-evergreen-tools" is a symbolic link to the other repository called drivers-evergreen-tools . That’s why it's become a broken link in the installed source code directory.

So, I checked the commit log in the mongo package repository and the file was added on 9 Sep 2020. And then, I saw the list of the mongo package’s version on the gems site. And it said as follows.

12.14.0 - December 01, 2020 (886KB)
22.14.0.rc1 - October 09, 2020 (883KB)
32.13.2 - December 01, 2020 (862KB)
42.13.1 - October 09, 2020 (855KB)
52.13.0 - July 30, 2020 (855KB)

So first, I deleted the installed directory and modified the version to 2.13.2 in the Gemfile as follows. And then, install the package again via "bundle install" command.

1gem 'mongo', '~> 2.13.2'

Luckily, the broken link had disappeared and succeeded to build by "sam build" command.

So, I’m not sure what’s OCSP exactly and why they put the link in the repository. However, I hope it will be helpful for somebody who has the same problems.

Ruby: bundle install hangs indefinitely with high CPU usage

Sun, 31 Jan 2021 00:08:05 +0900

Actually, I’m not a Ruby developer but I had an opportunity to help move some Ruby applications from EC2 to Lambda on AWS. At that time, I had an issue when I execute "bundle install" inside the container to build with the SAM framework. The problem was the command didn’t finish indefinitely. I searched about it and found some people who had the same issue but there was no way to work around it. I think changing or deleting versions will solve the problem. Let me share how to do that.

First, comment out all lines in the Gemfile, and then enable a package one by one. If you find the package which is the cause of the hanging, try the following way.

If it’s already specified some version, delete it.
Specify its latest version explicitly.
Specify the previous version of the latest one or the one after next

In my case, "activesupport" was the cause of the issue. Before I modified the Gemfile, it did have specified versions explicitly and installed "6.0.3.4" automatically by bundle. However, I checked rubygems.org and the latest version was "6.1.1" so I wrote it down like this and the problem was solved.

1gem 'activesupport', '~> 6.1.1'

I’m not sure this way will solve all the same issues but I hope it will be helpful for somebody who has the same problems.

Docker: What should I do when getting "503 service unavailable" error after executing docker pull command?

Sat, 26 Dec 2020 22:04:00 +0900

Getting "503 service unavailable" error after executing "docker pull" command is a very rare situation but there was in the past. So, it’s worth knowing for using docker. Most of the time, this is Docker Hub’ server side problem not client one. Once the error happens, we should visit their web site called Docker system status real-time view before struggling. When the error happened, may be "Docker Hub Registry" is in some trouble status.

Docker: How to solve Apache 403 Forbidden error?

Fri, 25 Dec 2020 01:28:00 +0900

It's a popular problem getting "403 Forbidden" errors from Apache. Even though it's working on the docker, it's the same but a little bit harder to investigate why it’s happened. However, most of the time, the cause of this problem is simple. Let’s take a look at some samples.

First, let me show you the correct way. Here is the vhost configuration file as vhost.conf.

1<VirtualHost *:80>
2    DocumentRoot /var/www/html
3</VirtualHost

And make a directory and put a text file for verifying.

1mkdir app/web
2echo "Hoge" > app/web/index.html

And make a docker-compose.yml file like this.

 1version: '3'
 2services:
 3  app:
 4    container_name: my-app
 5    image: php:7.4-apache
 6    ports:
 7      - 8080:80
 8    volumes:
 9      - ./app/web:/var/www/html
10      - ./vhost.conf:/etc/apache2/sites-enabled/000-default.conf

Then, it’s ready for running. Execute it like this.

1docker-compose up

And open a new terminal and execute curl command for verifying.

1$ curl localhost:8080
2Hoge

This is what I expected. It’s totally no problem. But sometimes "403 Forbidden" errors appear. How it happens? Most of the time, the cause of the error is modifying document root directory. Here is a sample.

Modify the document root setting in vhost.conf

1<VirtualHost *:80>
2    DocumentRoot /srv/app
3</VirtualHost>

And modify docker-compose.yml too.

 1version: '3'
 2services:
 3  app:
 4    container_name: my-app
 5    image: php:7.4-apache
 6    ports:
 7      - 8080:80
 8    volumes:
 9      - ./app/web:/srv/app
10      - ./vhost.conf:/etc/apache2/sites-enabled/000-default.conf

However, if I execute curl command the same as the last time, the error will be returned.

 1$ curl localhost:8080
 2<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 3<html><head>
 4<title>403 Forbidden</title>
 5</head><body>
 6<h1>Forbidden</h1>
 7<p>You don't have permission to access this resource.</p>
 8<hr>
 9<address>Apache/2.4.38 (Debian) Server at localhost Port 8080</address>
10</body></html>

The hint is Apache's default settings.

 1$ docker run -it --rm --entrypoint sh php:7.4-apache -c "cat /etc/apache2/apache2.conf"
 2  :
 3  :
 4<Directory />
 5        Options FollowSymLinks
 6        AllowOverride None
 7        Require all denied
 8</Directory>
 9
10<Directory /usr/share>
11        AllowOverride None
12        Require all granted
13</Directory>
14
15<Directory /var/www/>
16        Options Indexes FollowSymLinks
17        AllowOverride None
18        Require all granted
19</Directory>
20  :
21  :

As you see above, the first setting is set "all denied" for the root directory. And then, "/usr/share" and "/var/www/" is granted. So, even though I modify the directory path in the vhost.conf and docker-compose.yml file, the modified directory is not granted. So, the response will be the error.

So, how to fix it when I need to modify the document root directory? It’s necessary to add a grant setting like this.

1<VirtualHost *:80>
2    DocumentRoot /srv/app/web
3
4    <Directory "/srv/app/web">
5        AllowOverride all
6        Require all granted
7    </Directory>
8</VirtualHost>

The error will disappear.

And also, mistakes of the mount point is also a popular cause of some problems. For example, confusing "sites-available" and "sites-enabled" directories. In the traditional way, add the configuration file to "sites-available" first, and then execute "a2ensite" command to make it enabled and then restart the Apache process with graceful option. However, when executing the Apache on docker, no need to do that. Just mount it to 000-default.conf file under the "sites-enabled." If it’s mounted under the "sites-available" as follows, it will not be loaded by Apache.

1    volumes:
2      - ./app/web:/srv/app
3      - ./vhost.conf:/etc/apache2/sites-available/000-default.conf

Docker Hub: Error 429 too many requests

Thu, 19 Nov 2020 14:17:00 +0900

One day, I realized that one of my CodePipeline was failed to build the docker image. I investigated what caused the error. And then, I found out the following error message when executing docker build command.

1ERROR: failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/php/manifests/sha256:7b1cd0d9e1922e96ad3bc9fc44880cf7247f5a22a4badfd25480bf759edf2804: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I remembered the news which I read before that Docker Hub got started to set some limits and I checked the link which is included in the above error.

Understanding Docker Hub Rate Limiting

However, I was confused because I’m working for a very small start-up company and the CodePipeline runs a few times a day so I couldn’t understand why I reached the limit. Then, I came out that it may be "docker login" is necessary because the CodePipeline environment is shared servers so anonymous access was added up together. So, I got a new log-in account for building image and added the following command before docker build command in the buildspec in the CodePipeline and then I solved the problem.

1docker login -u hoge-hoge -p fuga

Kubernetes: How many pods are available per node on AWS EKS?

Sat, 10 Oct 2020 16:51:00 +0900

When getting started to use AWS EKS for building Kubernetes clusters, we should consider the limitations of the number of IP addresses it depends on the instance size. Actually, I didn’t know about it because I just overlooked it on the official document. I know it was my mistake but it should be a blind spot for some people so I’ll share my experience.

After a few months since I got started to use our cluster, I realize that there’s some error logs of kubelet like this.

1RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to set up sandbox container "xxxxxx" network for pod "hoge-1602061920-snvck": NetworkPlugin cni failed to set up pod "hoge-1602061920-snvck_apps" network: add cmd: failed to assign an IP address to container

I investigated this problem and then I found out there’s some limitation of the pod’s number by EC2 instance’s restrictions. EC2 instances has the limitation of the number of ENI (Elastic Network Interface) and how many IP addresses it’s able to have. The limitation is defined depending on the instance size. For example, m5.Large instance is able to have 3 ENIs and 10 IP addresses for each ENI so totally 30 addresses. However the primary address of each ENI cannot be used for Pods so 27 addresses will be available for pods. Another example, m5.4xlarge instance is able to have 8 ENIs and 30 IP addresses for each ENI so 232 addresses will be available for pods.

I’m not sure it’s enough compared to the resources like CPUs and memories for most people. However it should be considered before using EKS especially the cluster size is not large yet. In my case, after reaching the limitation, I’m operating as follows.

Use NodeAffinity option to avoid running multiple pods on the same node and don’t increase the replica number over the number of nodes
When running batches as CronJobs, adjust the schedule to avoid overlapping

References

AWS > Documentation > Amazon EC2 > User Guide for Linux Instances > Elastic network interfaces

HAProxy running on Kubernetes and output a lot of BADREQ logs

Fri, 09 Oct 2020 23:35:00 +0900

When I was setting up HAProxy running on our Kubernetes cluster, I defined the log format as JSON and I got a lot of BADREQ logs with status code 400 as follows.

 1{
 2  "conn": {
 3    "act": 7,
 4    "fe": 7,
 5    "be": 0,
 6    "srv": 0
 7  },
 8  "queue": {
 9    "backend": 0,
10    "srv": 0
11  },
12  "time": {
13    "tq": -1,
14    "tw": -1,
15    "tc": -1,
16    "tr": -1,
17    "tt": 0
18  },
19  "termination_state": "CR--",
20  "retries": 0,
21  "network": {
22    "client_ip": "172.31.79.179",
23    "client_port": 63740,
24    "frontend_ip": "172.31.76.184",
25    "frontend_port": 80
26  },
27  "ssl": {
28    "version": "-",
29    "ciphers": "-"
30  },
31  "request": {
32    "method": "<BADREQ>",
33    "uri": "-",
34    "protocol": "<BADREQ>",
35    "header": {
36      "host": "-",
37      "xforwardfor": "-",
38      "referer": "-"
39    }
40  },
41  "name": {
42    "backend": "https-in",
43    "frontend": "https-in",
44    "server": "<NOSRV>"
45  },
46  "response": {
47    "status_code": 400,
48    "header": {
49      "xrequestid": "-"
50    }
51  },
52  "bytes": {
53    "uploaded": 0,
54    "read": 245
55  }
56}

As a result of researching these error logs, I realize that the cause of these errors is the liveness probe of Kubernetes like this.

1        livenessProbe:
2          tcpSocket:
3            port: 80
4          initialDelaySeconds: 15
5          periodSeconds: 5

I’m not sure but I think health-check by TCP not HTTP will raise the same logs.

Finally, I solved this problem by adding "dontlognull" option like this.

1defaults
2    mode http
3    option dontlognull
4       :
5       :