gitlab lftp sftp

Deploy to sFTP from Gitlab CI

with LFTP

Friday, March 6, 2020

Using GitLab CI is a very convenient way to automate deployments of a website or blog. Most of the time it just works but sometimes it takes a bit more time to get things going.

The challenge I faced was the automatic deployment from Gitlab to a webhoster that only supports sFTP.

There are several workflows possible, I prefer to have just the source of the site in Gitlab and let the Gitlab CI take care of the rest: Workflow diagram

I came across an article by Savjee that used lftp to transfer the files with Gitlab CI to an sFTP host. In his article Savjee uses Ubuntu 18.04 as base image. For my project an MkDocs deployment, I used an Alpine image with Python (python:3.8.1-alpine3.11). That’s where the journey started.

Set environment variables in Gitlab

It’s good practice to not share your credentials for the sFTP server with the rest of the world therefore you use the Gitlab CI environment variables. The variables are created under Settings - CI / CD - Variables of the Gitlab project.

  • HOST, name of the sFTP server [Protected]
  • USER, username to login [Protected]
  • PASSWORD, password for the user [Protected & Masked]

Gitlab Variables

After setting the environment variables it took several attempts, each time solving another problem.

For the final configuration you skip to the final configuration. If you are interested in the full journey, continue reading.

Attempt 1: Modify .gitlab-ci.yml

Install LFTP

In the before_script section we install lftp.

1image: python:3.8.1-alpine3.11
2
3before_script:
4  - apk add git
5  - apk add --no-cache lftp
6...

Calling the lftp command in .gitlab-ci.yml

The lftp command is called from the deploy section in the .gitlab-ci.yml.

 1...
 2deploy:
 3  script:
 4    - mkdocs build
 5    - lftp -e "mirror -e --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD sftp://$HOST -p 22
 6  artifacts:
 7    paths:
 8      - site
 9  only:
10    - master

A breakdown of the lftp command lftp -e "mirror -e --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD sftp://$HOST -p 22

  • [-e command], the commands are seperated by a semicolon
    • mirror -e –transfer-all –reverse -X .* –verbose site/ www/
    • quit
  • [-u user[,pass]] [site], the user credentials and the url to the host
  • [-p port], the port to connect

sftp:auto-confirm yes:

  • lftp answers ‘yes’ to all ssh questions, in particular to the question about a new host key. Otherwise it answers ‘no’.

mirror -e --reverse -X .* --verbose site/ www/:

  • mirror [OPTS] [source [target]], mirror the specified source directory to the target directory.
    • [-e], delete files not present at the source
    • [–reverse], reverse mirror (send files)
    • [-X .*], exclude files starting with a dot ('.').
    • [–verbose], verbose level, default 0, no output
    • [site/ www/], source and destination

Result attempt 1

The Pipelline logging showed the LFTP command as last line. No error message just a slowly moving cursor waiting for …?

Attempt 2: Changing the LFTP command

Not sure what was going on, I asked my hosting provider for help. They suggested some changes to the LFTP command.

 1...
 2deploy:
 3  script:
 4    - mkdocs build
 5    - lftp -e "set sftp:auto-confirm yes; mirror -e --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD sftp://$HOST -p 22
 6  artifacts:
 7    paths:
 8      - site
 9  only:
10    - master

Changes to the LFTP command:

sftp:auto-confirm yes:

  • lftp answers ‘yes’ to all ssh questions, in particular to the question about a new host key.

mirror -e --transfer-all --reverse -X .* --verbose site/ www/:

  • mirror [OPTS] [source [target]], mirror the specified source directory to the target directory.
    • [–transfer-all], transfer all files, allways send all files

Result attempt 2

The same as in the first attempt, the LFTP command displayed as last line and a slowly moving cursor. Again waiting for …?

Attempt 3: Turn on debugging

In an attempt to get more information I turned on debugging (-d) on the LFTP command. Something I should have done earlier.

 1...
 2deploy:
 3  script:
 4    - mkdocs build
 5    - lftp -d -e "set sftp:auto-confirm yes; mirror -e --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD sftp://$HOST -p 22
 6  artifacts:
 7    paths:
 8      - site
 9  only:
10    - master

Result attempt 3

This time the result showed something very unexpected:

$ lftp -d -e "set cmd:trace yes;set sftp:auto-confirm yes; mirror -e --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD $HOST -p 22
 + set sftp:auto-confirm yes
 ---- Running connect program (ssh -a -x -s -l tisgoud.nl -p 22 tisgoud.nl.sftpurl.nl sftp)
 ---> sending a packet, length=5, type=1(INIT), id=0
 <--- sh: ssh: not found
 **** pty read: pseudo-tty: I/O error

<— sh: ssh: not found, SSH is not installed on the Alpine image 🤯!

LFTP

LFTP is a sophisticated file transfer program with command line interface. It supports FTP, HTTP, FISH, SFTP, HTTPS and FTPS protocols. GNU Readline library is used for input. Even the BitTorrent protocol is supported as built-in `torrent' command.

LFTP has mirror builtin which can download or update a whole directory tree. There is also reverse mirror (mirror -R) which uploads or updates a directory tree on server.

SFTP

SFTP (SSH File Transfer Protocol) is a file transfer protocol built upon the SSH transport layer and is used to securely move large amounts of data over an internet connection.

SFTP utilizes the SSH transport layer to establish a secure authenticated connection and provide organizations with a higher level of file transfer protection. It uses the SSH authentication and cryptographic capabilities to keep files safe during the transfer process.

Attempt 4: Add SSH

I added SSH to the before_script_part and left the LFTP command as it was.

 1image: python:3.8.1-alpine3.11
 2
 3before_script:
 4  - apk add git
 5  - apk add openssh
 6  - apk add --no-cache lftp
 7  - pip install mkdocs
 8  ...
 9
10  ...
11
12deploy:
13  script:
14    - mkdocs build
15    - lftp -d -e "set sftp:auto-confirm yes; mirror -e --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD sftp://$HOST -p 22
16  artifacts:
17    paths:
18      - site
19  only:
20    - master

Result attempt 4

$ lftp -d -e "set cmd:trace yes;set sftp:auto-confirm yes; mirror -e --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD $HOST -p 22
 + set sftp:auto-confirm yes
 ---- Running connect program (ssh -a -x -s -l tisgoud.nl -p 22 tisgoud.nl.sftpurl.nl sftp)
 ---> sending a packet, length=5, type=1(INIT), id=0
 The authenticity of host 'tisgoud.nl.sftpurl.nl (148.209.164.253)' can't be established.
 <--- RSA key fingerprint is SHA256:dGoG9cfrjgozdgG65ZNLK/dHygnqJxq+JCZm/1AbTwU.
 **** Timeout - reconnecting
 ---- Disconnecting

SSH is working but now the auto-confirm option is not working. The host is not automatically added to the list of known hosts.

Attempt 5: Add host to known_hosts

Since the auto-confirm is not working we will try to add the host manually with the following command in the deploy section:

ssh-keyscan -H $HOST -p 22 >> /root/.ssh/known_hosts

Result attempt 5

$ ssh-keyscan -H $SERVER >> ~/.ssh/known_hosts
 /bin/sh: eval: line 106: can't create /root/.ssh/known_hosts: nonexistent directory
 ERROR: Job failed: exit code 1

So ssh-keyscan is unable to create the directory /root/.ssh.

Attempt 6: Create known_hosts file

We create the directory root/.ssh and the empty file known_hosts before we call ‘ssh-keygen’ and set the right permissions.

 1image: python:3.8.1-alpine3.11
 2
 3before_script:
 4  - apk add git
 5  - apk add openssh
 6  - apk add --no-cache lftp
 7  - pip install mkdocs
 8  ...
 9
10  ...
11
12deploy:
13  script:
14    - mkdocs build
15    - mkdir /root/.ssh
16    - chmod 700 /root/.ssh
17    - touch /root/.ssh/known_hosts
18    - chmod 600 /root/.ssh/known_hosts
19    - ssh-keyscan -H $HOST >> /root/.ssh/known_hosts
20    - lftp -e "set sftp:auto-confirm yes; mirror -e --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD sftp://$HOST -p 22
21  artifacts:
22    paths:
23      - site
24  only:
25    - master

Result step 6

Yay, LFTP is sending files.

$ ssh-keyscan -H $SERVER >> /root/.ssh/known_hosts
# tisgoud.nl.sftpurl.nl:22 SSH-2.0-SshReverseProxy
# tisgoud.nl.sftpurl.nl:22 SSH-2.0-SshReverseProxy
# tisgoud.nl.sftpurl.nl:22 SSH-2.0-SshReverseProxy
$ lftp -d -e "set sftp:auto-confirm yes; mirror -e --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD $HOST -p 22
Removing old file `404.html'
Transferring file `404.html'
...

The final configuration

With Savjee’s post as base until the working configuration the following items changed:

  • Installed LFTP (lftp packaged)
  • Installed SSH (openssh package)
  • Created the known_hosts file
  • Added the HOST to the known_hosts file

Two additional steps:

  • Removed the auto-confirm option from the LFTP command (cleaning up)
  • Added -P 4 option to the LFTP command to speed things up (parallel up/downloads)

The full .gitlab-ci.yml:

 1image: python:3.8.1-alpine3.11
 2
 3before_script:
 4  - apk add git
 5  - apk add openssh
 6  - apk add --no-cache lftp
 7  - pip install mkdocs
 8  # Add your custom theme if not inside a theme_dir
 9  # (https://github.com/mkdocs/mkdocs/wiki/MkDocs-Themes)
10  - pip install mkdocs-material
11  # Add your plugins
12  - pip install pygments
13  - pip install pymdown-extensions
14  - pip install mkdocs-awesome-pages-plugin
15  - pip install pyembed-markdown
16  - pip install mkdocs-git-revision-date-localized-plugin
17  - pip install plantuml-markdown
18  - pip install mkdocs-minify-plugin
19
20deploy:
21  script:
22    - mkdocs build
23    - mkdir /root/.ssh                                # Create .ssh directory
24    - chmod 700 /root/.ssh                            # Set permissions
25    - touch /root/.ssh/known_hosts                    # Create known_hosts files
26    - chmod 600 /root/.ssh/known_hosts                # Set permissions
27    - ssh-keyscan -H $HOST >> /root/.ssh/known_hosts  # Add server to known_hosts, "set sftp:auto-confirm" does not work
28    - lftp -e "mirror -e -P 4 --transfer-all --reverse -X .* --verbose site/ www/; quit" -u $USER,$PASSWORD sftp://$HOST -p 22
29  artifacts:
30    paths:
31      - site
32  only:
33    - master

Lessons learned

From the initial article to the final configuration there were only three real differences. It took way more time than initialy intended. I keep thinking what I could have done differently.

My lessons learned:

  • Copy & Paste code from an article is easy until it doesn’t work
  • Take time to learn more about the used tooling
  • Turn on debugging and verbosity as soon as you run into problems
  • Share your experiences to prevent others failing

ToC