My thoughts on Slackware, life and everything

Month: February 2022 (Page 1 of 2)

Slackware Cloud Server Series Episode 6: Etherpad with Whiteboard

Hi all!
This is the 6th episode in a series I am writing about using Slackware as your private/personal ‘cloud server’. It is an unscheduled break-out topic to discuss an Etherpad server specifically.
Check out the list below which shows past, present and future episodes in the series, if the article has already been written you’ll be able to click on the subject.
The first episode also contains an introduction with some more detail about what you can expect.
These articles are living documents, i.e. based on readers’ feedback I may add, update or modify their content.

Etherpad with Whiteboard

In Episode 3 (Video Conferencing) we setup a Jitsi Meet server in a Docker container stack which includes an Etherpad server for real-time document collaboration during a video meeting.
That Etherpad instance as configured by the Docker-Jitsi-Meet project is really only a demo setup. It uses a “dirtydb” JSON backend wich is not meant for anything else but testing. It really needs a proper SQL database like MariaDB to power it. And you can’t export your documents from this demo Etherpad in any meaningful format.
Furthermore, this Etherpad container is not using our Keycloak IAM for authentication; everyone who knows the public URL can create a document, invite others and start writing. Even shared documents created in Jitsi meetings are not secure and anyone who guesses the room name has access to the Etherpad document.

This article means to set things right and configure Etherpad correctly, adding Whiteboard functionality as we go. I will also discuss the differences between our Jitsi integrated Etherpad and a running a standalone Etherpad server in case you are not interested in Video meetings and only want the text collaboration.

Preamble

This article assumes you have already setup an Etherpad in a Docker container as part of a Dockerized Jitsi Meet server (see Episode 3 in this series), and this Etherpad is running at a publicly accessible URL:

  • https://meet.darkstar.lan/pad/

We want to make it delegate user authentication to our OpenID Provider: Keycloak. That Keycloak service is available at:

  • https://sso.darkstar.lan/auth

If you are not interested in Jitsi Meet and only want to know how to run an Etherpad server, this article still contains everything you need but keep in mind that my examples are all assuming the above URL for the Etherpad. Adapt that URL to your own real-life situation. You may still have to setup an Apache webserver first, which serves an empty page at “https://meet.darkstar.lan/” but I will leave that to you.

Configuring MariaDB

By default, Etherpad will use a ‘DirtyDB’ JSON file-based backend. It is straight-forward to make it switch to for instance a MariaDB database server backend, we only need to provide the connection details for a pre-existing database.
Like with the previous articles we are using the Slackware MariaDB database server which is running on the host. First, we will create a database (etherpad_db), a database user (etherpad) and grant this user sufficient access to the database. Then we will use these database configuration values when editing the Docker-Jitsi-Meet files in order to change the Etherpad container properties.
This is how we create the database and the user (using a secure password string for ‘EPPPASSWD‘ of course):

$ mysql -uroot -p
> CREATE DATABASE IF NOT EXISTS `etherpad_db` CHARACTER SET utf8 COLLATE utf8_unicode_ci;
> CREATE USER 'etherpad'@'localhost' identified by 'EPPASSWD';
> CREATE USER 'etherpad'@'%' identified by 'EPPPASSWD';
> GRANT CREATE,ALTER,SELECT,INSERT,UPDATE,DELETE on `etherpad_db`.* to 'etherpad'@'localhost';
> GRANT CREATE,ALTER,SELECT,INSERT,UPDATE,DELETE on `etherpad_db`.* to 'etherpad'@'%';
> FLUSH PRIVILEGES;
> exit;

Note from the above SQL statements that we are allowing the ‘etherpad‘ user remote access to the database. This is needed because Etherpad in the Docker container contacts MariaDB via the network, using the IP address of the Docker network bridge in the Jitsi Meet container stack.

Reconfiguring Docker-Jitsi-Meet

My advise is to start with briefly re-visiting Episode 3 of the series and read back how we customized the ‘docker-compose.yml‘ and ‘.env‘ files in order to startup the Docker-Jitsi-Meet stack properly. Because we are going to update these two files again.
This is what we need to change to make Etherpad connect to the external MariaDB database:

Relevant .env additions:

# MariaDB parameters for mysql DB instead of dirtydb
ETHERPAD_DB_TYPE=mysql
ETHERPAD_DB_HOST=172.20.0.1
ETHERPAD_DB_PORT=3306
ETHERPAD_DB_NAME=etherpad_db
ETHERPAD_DB_USER=etherpad
ETHERPAD_DB_PASS=EPPASSWD
ETHERPAD_DB_CHARSET=utf8

Relevant docker-compose additions:

In the ‘.env‘ file we defined the IP address for the database server (172.20.0.1). Etherpad is running inside a container, and its way out is through the default gateway of its Docker network. In order to have 172.20.0.1 as the gateway address, we need to configure the internal ‘meet.jitsi‘ network a deterministic IP range so that we always know its gateway address. if we are going to give that network the IP range “172.20.0.0/16“, the “networks” statement all the way at the bottom needs to be changed from:

# Custom network so all services can communicate using a FQDN
networks:
  meet.jitsi:

to:

# Custom network so all services can communicate using a FQDN 
networks: 
  meet.jitsi: 
    ipam: 
      config: 
        - subnet: 172.20.0.0/16

Use the variables we added to ‘.env’ to create an updated Etherpad container definition. Right underneath this line:

            - SKIN_VARIANTS=${ETHERPAD_SKIN_VARIANTS}

Add the following lines:

            - DB_TYPE=${ETHERPAD_DB_TYPE} 
            - DB_HOST=${ETHERPAD_DB_HOST} 
            - DB_PORT=${ETHERPAD_DB_PORT} 
            - DB_NAME=${ETHERPAD_DB_NAME} 
            - DB_USER=${ETHERPAD_DB_USER} 
            - DB_PASS=${ETHERPAD_DB_PASS} 
            - DB_CHARSET=${ETHERPAD_DB_CHARSET} 

Accessing the admin console

Etherpad has an admin console where you can manage its plugin configuration and other things too. It will only be enabled if you configure an admin password. So let’s do that too.
This is what we need to change to enable the admin console for Etherpad:

Relevant .env additions:

# The password for Etherpad admin page
ETHERPAD_ADMIN_PASSWORD="my_secret_admin_pass"

Relevant docker-compose additions:

Right underneath this line:

            - SKIN_VARIANTS=${ETHERPAD_SKIN_VARIANTS}

Add the following lines:

            - ADMIN_PASSWORD=${ETHERPAD_ADMIN_PASSWORD}

Relevant Apache httpd additions:

If you would now access the URL for the admin console, https://meet.darkstar.lan/pad/admin/ you would only see the message “Unauthorized“. The Etherpad expects you to provide the Basic Authentication hook in front of that page which passes the admin credentials on to the backend. So, we will add a ‘AuthType Basic‘ block to our Apache httpd configuration to add Basic Authentication which will pop up a login dialog, and then add the admin user and its password “my_secret_admin_pass” to a htaccess file.

Remember, in Episode 4 we configured the Etherpad to be available at “https://meet.darkstar.lan/pad/” which means the admin console URL is “https://meet.darkstar.lan/pad/admin/
This is the block to add to your VirtualHost configuration for the Etherpad:

<Location /pad/admin>
    AuthType Basic
    AuthBasicAuthoritative off
    AuthName "Welcome to the Etherpad"
    AuthUserFile /etc/httpd/passwords/htaccess.epl
    Require valid-user
    Order Deny,Allow
    Deny from all
    Satisfy Any
</Location>

And then we still need to create that htaccess file using the ‘htpasswd‘ tool, like this:

# mkdir /etc/httpd/passwords
# htpasswd -B -c /etc/httpd/passwords/htaccess.epl admin

The “-B” parameter enforces the use of bcrypt encryption for passwords. This is currently considered to be very secure.
The above command will prompt for the password, and there you enter that “my_secret_admin_pass” string. The content of that file will look like this:

# cat /etc/httpd/passwords/htaccess.epl
admin:$2y$05$JpvucTlKIQEJViCynem.JelENHpv/maJStsPM4iU9d/sg4cMU.UfW

The Docker Jitsi Meet container stack needs to be refreshed and restarted since we edited ‘.env’. I don’t want to repeat the detailed instructions here, so refer you to the section “Considerations about the “.env” file” in Episode 3 of this article series. Do that now, and when the updated container stack is up and running again, continue here.

And after also restarting the Apache httpd and refreshing the URL “https://meet.darkstar.lan/pad/admin/” you will be asked to enter your admin credentials and you will end up in the Etherpad admin console.
The screenshot below does not reflect the status of the barebones Etherpad by the way; you see a lot of installed plugins mentioned on the admin page. We will be installing those into the Etherpad image in one of the next sections:

At this stage, we have accomplished a well-performing Etherpad installation with a SQL database back-end and an administrative web-interface. The next step is to add authentication through an OpenID provider like our Keycloak IAM server.

Integrating with Keycloak IAM

The out-of-the-box Etherpad Docker container is not very functional. The above sections already showed how to replace the “DirtyDB” with a proper SQL database server like MariaDB. But the default image misses a few useful plugins and a real desktop editor program which allows Etherpad users to export their collaborative work to a proper document format instead of pure HTML.

Whatever plugins we add, at the very least we need to add a plugin which allows us to let the Etherpad authenticate against our Keycloak IAM server. This plugin needs to be inside the Docker image, we cannot use it outside the running container. There’s no other option than to create a custom Docker image for Etherpad. We use this as an opportunity to add some more plugins, as well as Abiword (to enable document export in Etherpad).

I’ll show how to announce Etherpad to Keycloak (we create a Client profile in Keycloak); then I’ll share the required configuration to be added to the Etherpad Docker files; and then I’ll show how to create a custom Docker image enriched with additional plugins which we will use instead of the basic image from Docker Hub.

Keycloak

First of all, let’s create a Client profile for Etherpad in Keycloak.

  • Login to the Keycloak Admin Console (https://sso.darkstar.lan/auth/admin/)
    • Select our ‘Foundation‘ realm from the dropdown at the left.
    • Under ‘Clients‘, create a new client:
      ‘Client ID’ = “etherpad
      ‘Root URL’ = “https://meet.darkstar.lan/pad/ep_openid_connect/callback
      Note that for the Etherpad OIDC plugin ‘ep_openid_connect’  – see below – to work, the ‘Valid Redirect URIs’  (a.k.a. callback URL) must be the concatenation of the Etherpad base URL (https://meet.darkstar.lan/pad/) plus “/ep_openid_connect/callback“. When setting the ‘Root URL‘ to the above value, the ‘Redirect URIs‘ will automatically also be set correctly to “https://meet.darkstar.lan/pad/ep_openid_connect/callback/*
    • Save.
    • In the ‘Settings‘ tab, change:
      ‘Access Type’ = “confidential” (default is “public”)
    • Save.
    • Go to the ‘Credentials‘ tab
    • Make sure that ‘Client Authenticator‘ is set to “Client Id and Secret
    • Copy the value of the ‘Secret‘, which we will use later in the Etherpad connector; the Secret will look somewhat like this:
      2jnc8H6RH9jIYMXExUHA7XF7uD8YKIRs“.

Keycloak configuration being completed, we can turn our attention to the connector between Etherpad and Keycloak.

EP_openid_connect

We will use the Etherpad plugin ep_openid_connect which I already briefly mentioned earlier. This plugin provides the needed OpenID client functionality to Etherpad.
When we add this plugin to the Etherpad Docker image we need to be able to configure it via the ‘docker-compose.yml‘ and ‘.env‘ files of Docker-Jitsi-Meet. The existing configuration files in the repository for the docker-jitsi-meet stack are just meant to make the basic Etherpad work, so we need to add more parameters to configure our custom Etherpad properly.
I will show you what you need to add, and where.

The ‘ep_openid_connect’ plugin expects an ‘ep_openid_connect’ block in the ‘settings.json’ file (we will get to that file in the next section). Since that file is JSON-formatted, we arrive at the following structure:

"ep_openid_connect": {
"issuer": "https://sso.darkstar.lan/auth/realms/foundation",
"client_id": "etherpad",
"client_secret": "2jnc8H6RH9jIYMXExUHA7XF7uD8YKIRs",
"base_url": "https://meet.darkstar.lan/pad"
},

The text string values in green highlight are of course the relevant ones. What is the meaning of the parameters:

  • issuer: this is the string you obtain through Keycloak’s OpenID Discovery URL. Make sure you have ‘jq‘ installed and then run this command to obtain the value for ‘issuer‘:
    $ curl https://sso.darkstar.lan/auth/realms/foundation/.well-known/openid-configuration | jq .issuer
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  5749  100  5749    0     0   6594      0 --:--:-- --:--:-- --:--:--  6600
    "https://sso.darkstar.lan/auth/realms/foundation"

    In this case, you could easily have guessed the ‘issuer’ value, but using the above ‘well-known’ query URL will always get you the correct value.

  • client_id, client_secret: those are the same OAuth2 values obtained from Keycloak when creating the Etherpad Client profile as seen above.
  • base_url: this is the URL where Etherpad is externally accessible (https://meet.darkstar.lan/pad/ – see Episode 3).

Additionally, since we are now enforcing login, Etherpad’s ‘requireAuthentication‘ setting must be set to “true”. Note that the default setting is “false”; this is how that setting is defined in the Etherpad configuration:

"requireAuthentication": "${REQUIRE_AUTHENTICATION:false}",

We’ll just have to define a “true” value for that variable later on.

Note: Each configuration parameter can also be set via an environment variable, using the syntax "${ENV_VAR}" or "${ENV_VAR:default_value}". This ability is what we will use when updating the Docker Compose file for Jitsi Meet. We will not use the literal JSON block above, instead we will fill it with variable names and use our Docker Compose files to provide values for these variables. That way I am able to create a generic Docker image that I can upload to the Docker Hub and share with other people.
The file ‘settings.json.template‘ in the Etherpad repository has lots of examples.

Hold on to that thought for a minute while we proceed with creating our custom Etherpad Docker image, since we have all the data available to do this now. Once we have that image, we will once again return to the re-configuration of Docker Jitsi Meet and integrate our Etherpad with the Jitsi container stack.

Custom Etherpad Docker image

How to create a custom Docker image?

  • First we clone the “etherpad-lite” git repository. That contains a Dockerfile plus all the context that is needed by ‘docker build‘ to generate an image.
    $ mkdir ~/docker-etherpad-slack
    $ cd ~/docker-etherpad-slack
    $ git clone https://github.com/ether/etherpad-lite .
  • There is one relevant configuration file in the root directory of the checked-out repository:  ‘settings.json.docker‘. This file will be copied into the Etherpad Docker image and renamed to ‘settings.json‘ when we run ‘docker build‘ command. Any plugin configuration we want to enable via environment variables needs to be present in this file.
    Now the standard configurable parameters for the Etherpad are contained in that file, but our custom settings for the “ep_openid_connect” plugin are not. I already showed you how that block of configurable parameters looks in the previous section, and I promised to parametrize it. This is how the parameters look, and we will give them values in the next section where we update the Docker Jitsi Meet configuration.
  • Relevant ‘settings.json.docker’ additions:
    • Support for OpenID Connect in Etherpad – add this JSON code:
      "ep_openid_connect": {
      "issuer": "${OIDC_ISSUER:undefined}",
      "client_id": "${OIDC_CLIENT_ID:undefined}",
      "client_secret": "${OIDC_CLIENT_SECRET:undefined}",
      "base_url": "${OIDC_BASE_URL:undefined}"
      },
    • We also add a connector for the WBO Whiteboard server (its setup is described in the next section below) to the Docker image: the plugin is called ‘ep_whiteboard‘ and needs the following JSON configuration block to be added:
      "ep_draw": {
      "host": "${WBO_HOST:undefined}"
      },
    • Enable AbiWord in the configuration, since we are going to add it to the image. The full path to the ‘abiword‘ binary needs to be configured in ‘settings.json.docker‘.
      Look up this line in the file:
      "abiword": "${ABIWORD:null}",
      and change it to:
      "abiword": "${ABIWORD:/usr/bin/abiword}",
  • Then we build a new image, adding several useful (according to the developers) plugins, as well as the Abiword word processor.
    I tag the resulting image as “liveslak/etherpad” so that I can upload (push) it to the Docker Hub later on:

    $ docker build \
    --build-arg ETHERPAD_PLUGINS="ep_openid_connect ep_whiteboard ep_author_neat ep_headings2 ep_markdown ep_comments_page ep_align ep_font_color ep_webrtc ep_embedded_hyperlinks2" \
    --build-arg INSTALL_ABIWORD="yes" \
    --tag liveslak/etherpad .

This leads to the following output and results in an image which is quite a bit larger (786 MB uncompressed) as the standard Etherpad image (474 MB uncompressed) because of the added functionality:

Step 24/26 : HEALTHCHECK --interval=20s --timeout=3s CMD ["etherpad-healthcheck"]
---> Running in 2c241a795e46
Removing intermediate container 2c241a795e46
---> 5ca0246c1e61
Step 25/26 : EXPOSE 9001
---> Running in 21fcdf511d46
Removing intermediate container 21fcdf511d46
---> 1c288502f632
Step 26/26 : CMD ["etherpad"]
---> Running in 08a25b585280
Removing intermediate container 08a25b585280
---> 912c54fb6c0a
Successfully built 912c54fb6c0a
Successfully tagged liveslak/etherpad:latest

If you create the image on another computer and need to transfer it to your Slackware Cloud Server in order to use it there, you can save the image to a compressed tarball on the build machine, using docker commands:
$ docker save liveslak/etherpad | xz > etherpad-slack.tar.xz

You can use ‘rsync’ or ‘scp’ to transfer that tarball to your Cloud Server and then load it into the Docker environment there, also using docker commands so that you don’t need to know the intimate details on how Docker works with images:
$ cat etherpad-slack.tar.xz | docker load

I pushed this image to my own Docker repository https://hub.docker.com/repository/docker/liveslak/etherpad but I first added a tag to reflect the latest Etherpad release (1.8.16 at the moment):

$ docker login
$ docker tag liveslak/etherpad liveslak/etherpad:1.8.16
$ docker push liveslak/etherpad:1.8.16
$ docker push liveslak/etherpad:latest

This means that you can use the Hub version of ‘liveslak/etherpad’. But you can just as well use your own locally generated etherpad image in the ‘docker run‘ commands that launch your Etherpad container.
When you have a local image called “liveslak/etherpad”, then Docker will not check for an online image called “liveslak/etherpad”. If you did not generate your own image, Docker will look for (and find) my image at the Hub (or at the private Registry you may have configured), so it will download and use that.

Setting up WBO Whiteboard

Etherpad will be even more attractive if it offers users a collaborative Whiteboard and not just a collaborative text editor.
Enter WBO, which is an actual drawing board with infinite canvas and real-time refresh for all users.
Its boards are persistent; if you re-visit a board later on, all your content will still be there. Look at the WBO demo site… amazing.

We will run WBO in its own Docker container and re-configure our Etherpad webserver with a reverse proxy so that WBO can be integrated into Etherpad through the ‘ep_whiteboard’ connector.
It’s not so complex actually.

Docker container

First, launch a Docker container running WBO. We ensure that the data of the whiteboards you will be creating are going to be stored persistently outside of the container, so let’s create that data directory first and ensure that the internal WBO user is able to write there (you may have a different preference for directory location):

# mkdir -p /opt/dockerfiles/wbo-boards
# chown 1000:10 /opt/dockerfiles/wbo-boards

Then launch the container as a background process, and make it listen at port “5001” of your host’s loopback address:

$  docker run -d -p localhost:5001:80 -v "/opt/dockerfiles/wbo-boards:/opt/app/server-data" --restart unless-stopped --name whiteboard lovasoa/wbo:latest

Reverse proxy

We make WBO available behind an apache httpd reverse proxy which takes care of the encryption (https) using a Let’s Encrypt certificate.

Add the following block to your <VirtualHost></VirtualHost> definition of the server which also defines the reverse proxy for your Etherpad (which is https://meet.darkstar.lan/pad/ remember?):

# Reverse proxy for the WBO whiteboard Docker container:
<Location /whitepad/>
    ProxyPass http://127.0.0.1:5001/
    ProxyPassReverse http://127.0.0.1:5001/
</Location>

After restarting Apache httpd, your WBO whiteboard will be accessible via https://meet.darkstar.lan/whitepad/ . We will use that green highlighted text down below as the value for the ETHERPAD_WBO_HOST variable. Etherpad will prefix that text with “https://” and that prefix cannot be changed… hence the requirement for a reverse proxy that can handle the data encryption.

One caveat when you do this on your real-life internet-facing cloud server…
The Whiteboard server is accessible without authentication. It may be advisable to just come up with a different path component than “/whitepad/“, you can think of something like a UUID-like string: “/8cd77cbe-a694-4390-800a-638c7cc05f49/” as long as you use the same string in both places (reverse proxy and ETHERPAD_WBO_HOST definitions). Also, your board names are not visible anywhere unless you share their URLS with other people. So, a relatively safe environment.

Using the custom Etherpad with Jitsi Meet

If you followed Episode 3, you will have a directory “/usr/local/docker-jitsi-meet-stable-6826“, your version number may differ from my “6826“. Inside you will have your modified ‘docker-compose.yml‘ file.

We are going to edit two files: ‘.env‘ and ‘docker-compose.yml‘.

  • Relevant ‘.env’ additions:
    In the ‘.env‘ file we define correct values for the variables we introduced earlier. You can add the following lines basically anywhere, but it is of course most readable if you copy them immediately after the other ETHERPAD_* variables you added earlier on for the MySQL database backend:
    ETHERPAD_OIDC_ISSUER="https://sso.darkstar.lan/auth/realms/foundation"
    ETHERPAD_OIDC_BASE_URL="https://meet.darkstar.lan/pad/"
    ETHERPAD_OIDC_CLIENT_ID="etherpad"
    ETHERPAD_OIDC_CLIENT_SECRET="2jnc8H6RH9jIYMXExUHA7XF7uD8YKIRs"
    ETHERPAD_REQUIRE_AUTHENTICATION="true"
    ETHERPAD_WBO_HOST="meet.darkstar.lan/whitepad"
  • Relevant ‘docker-compose.yml’ additions:
    Add the following lines to the “etherpad:” section immediately below the MySQL database variable definitions you added earlier on in this Episode. You notice the variable names we defined in the previous section when dealing with ‘ep_openid_connect‘:
    - OIDC_ISSUER=${ETHERPAD_OIDC_ISSUER}
    - OIDC_CLIENT_ID=${ETHERPAD_OIDC_CLIENT_ID}
    - OIDC_CLIENT_SECRET=${ETHERPAD_OIDC_CLIENT_SECRET}
    - OIDC_BASE_URL=${ETHERPAD_OIDC_BASE_URL}
    - REQUIRE_AUTHENTICATION=${ETHERPAD_REQUIRE_AUTHENTICATION}
    - WBO_HOST=${ETHERPAD_WBO_HOST}
  • More ‘docker-compose.yml’ updates:
    The “etherpad:” service definition in that YAML file contains the following reference to the Etherpad Docker image:

image: etherpad/etherpad:1.8.16

You need to change that line to:

image: liveslak/etherpad:1.8.16

…in order to use our custom Etherpad image instead of the default one.

The re-configuration is complete and since we modified the ‘.env‘ file again, we  need to refresh and restart our Docker Jitsi Meet container stack again.
Note however, that at this point we have to perform this restart differently than mentioned earlier in this article. Since we are switching to a new Etherpad image, the container based on the old image needs to be removed also. For this scenario, please consult the detailed instructions in both sections “Considerations about the “.env” file” and “Upgrading Docker-Jitsi-Meet” in Episode 3 of this article series.
The complete set of steps to follow is a mix of both sections, and I share it with you for completeness’ sake:

# cd /usr/local/docker-jitsi-meet-stable-*
# docker-compose down
# rm -rf /usr/share/docker/data/jitsi-meet-cfg/
# mkdir -p /usr/share/docker/data/jitsi-meet-cfg/{web/letsencrypt,transcripts,prosody/config,prosody/prosody-plugins-custom,jicofo,jvb,jigasi,jibri}
# docker-compose pull
# docker-compose up -d

Don’t forget to remove the old, unused, Etherpad image because it is now wasting 474 MB uncompressed disk space.

Summarizing

Even though we set it up as part of the Jitsi stack, we now have a standalone Etherpad running which requires you to login when you visit “https://meet.darkstar.lan/pad/”.
On the other hand, you can also access Etherpad via Jitsi Meet. What’s different?
When you start a Jitsi meeting, via “https://meet.darkstar.lan/” and then click on “Open shared document“, you are already authenticated against Keycloak and the Etherpad document will open for you right away, no second login required.

After login, you will be met with a much more powerful editor than the basic one that comes with Docker Jitsi Meet. You’ll notice the extended document export capability thanks to Abiword and the small video widget at the top for face-to-face communication thanks to the WebRTC plugin.

Happy collaborating!

Running the custom Etherpad standalone

If you are not interested in Jitsi Meet, this is the command to start the customized Etherpad container and make it listen at port 9001 of the loopback address:

# docker run -d -p 127.0.0.1:9001:9001 liveslak/etherpad

The Etherpad container is now accessible only on your computer by pointing your browser at http://localhost:9001/ . You still need to add an Apache reverse proxy definition to the VirtualHost site definition to make your Etherpad available for other users at https://meet.darkstar.lan/pad/ .
If you want to change the container’s behavior using the available variables as documented before, you can pass these to the ‘docker run‘ command using one or more “-e” parameters, like so (this example just enables the admin console):

# docker run -d -p 127.0.0.1:9001:9001 \
  -e ADMIN_PASSWORD="my_secret_admin_pass" \
  --name etherpad \
  liveslak/etherpad

With additional environment variables you can enable more of the latent functionality. See the earlier sections of this article for all the relevant variables: those that enable the MySQL database backend; the one that enables the Whiteboard; those that enable the Keycloak authentication, etc.

Thanks

Etherpad with integrated Whiteboard can be a compelling solution for some user groups. Even without Jitsi Meet, you can jointly write and draw, save your work to your local harddrive and you have voice & video in a small overlay if you need to discuss the proceedings.
I encourage you to try it out. With or without integration into Jitsi Meet or even without Keycloak authentication if you want to create this as a completely free and low-treshold service to your local community.

Let me know what you think of this Episode in the comments section below. The final Episode, how to setup your own private Docker image repository, will take some time to write… I have not yet started doing in-depth research on that topic. But the six available Episodes will hopefully keep you occupied for a while 🙂
Thanks for reading until the end.

Eric

I now have a US mirror for Slackware Live and other goodies

Thanks to an anonymous sponsor, I am now operating a physical server in a US data center with a 1 Gbps connection to the Internet.

This server addresses a complaint of many people who are trying to download ISOs of the Slackware Live Edition. My slackware.nl aka download.liveslak.org server is hosted in a Dutch datacenter in Amsterdam, and it looks like people outside Europe, in particular downloaders in Southern Pacific region, are experiencing terribly slow speeds when fetching content from that server.

My new US server is available at two main URLs:

  • http://us.liveslak.org is the go-to location for all content related to Slackware Live Edition.
  • http://taper.alienbase.nl (don’t be confused by the “.nl” domain… I do not own a “.us” domain unfortunately) is the resurrection of my old “taper” VM which did not survive the original release of liveslak… that taper buckled under the high demand caused by massive download traffic and I decommisioned it in favor of my French datacenter server “bear” which again was replaced with “martin” in Amsterdam.
    The new taper has mirrors for liveslak (exact same content as us.liveslak.org) and also all Slackware release trees and ISOs, the ‘cumulative’ package repository, Mate SlackBuild (msb) and Cinnamon SlackBuild (csb), as well as my own package and multilib repositories.

In addition to the http access, these servers are also accessible via rsync: rsync://us.liveslak.org/ and rsync://taper.alienbase.nl/.

I hope this will give you folks out there a good alternative mirror location. Let me know how you experience the download speeds.

Cheers, Eric

Chromium security update remedies actively used exploit

New chromium and chromium-ungoogled packages for Slackware!
The recent Google Chromium update aims to plug a security hole which is already exploited out there, allowing attackers to take control of your computer. See CVE-2022-0609.
Get my Chromium packages for version 98.0.4758.102 (regular as well as un-googled) and upgrade to these as soon as you can: https://slackware.nl/people/alien/slackbuilds/chromium/ and https://slackware.nl/people/alien/slackbuilds/chromium-ungoogled/ .

These packages work on Slackware 14.2 and newer, 32bit as well as 64bit variants still of course.

Eric

Challenges with TigerVNC in Slackware 15.0

The 1.12.0 version of TigerVNC which is present in Slackware 15.0, is quite different from earlier versions such as the 1.6.0 version in Slackware 14.2 and even the previous iterations of this software in Slackware-current ( up to 1.11). It has ‘evolved‘ in a way that it has become dependent on systemd, or so the developers claim.

And indeed, the most prominent change is that the old ‘vncserver‘ script has been rewritten and should not be run directly any longer. In previous versions, as a user you could run ‘vncserver :1‘ to start a VNC server on port “5900 + 1” aka TCP Port 5901, and if needed you could kill that VNC server session with ‘vncserver -kill :1‘.
Fast forward to current-day. You are now expected to start the VNC server via the new command ‘vncsession‘ which will look in a couple of places to find out who you are and what desktop session you want to start. No longer will it install a “${HOME}/.vnc/xstartup” script for you to customize, instead it will look first in ‘/usr/share/xsessions‘ for *.desktop files like graphical login managers also do (SDDM, LightDM). Slackware applied a patch here for convenience, so that the names of sessions to look for also include "/etc/X11/xinit/Xsession", "/etc/X11/Xsession", "${HOME}/.vnc/xstartup", "${HOME}/.xinitrc", "/etc/X11/xinit/xinitrc" in that order.

The new TigerVNC expects to be launched from a systemd service and it can no longer be started as a non-root user.

Accepting that as a given (we can argue all we want with these developers but it looks that they are not interested), I looked for a way to make life easy for me and other VNC users on Slackware and other non-systemd distros.
In this article I will describe the solution I came up with. It’s a hack, I don’t think it is the best, but it works for me and only needs a one-time configuration by the root user. Let me know in the comments section how you were affected and dealt with the changes in TigerVNC!

  • The new TigerVNC uses a user-to-displayport mapping file, ‘/etc/tigervnc/vncserver.users‘ with lines that contain “port=user” mappings.
    For instance, a line containing “:9=alien” means that a VNC server session which is started for user “alien” will be running at VNC port “:9” which corresponds to TCP port 5909.
  • If a VNC session can only be started as root, then I will use ‘sudo‘ to allow regular users to start a ‘/usr/local/sbin/vncsession-start‘ wrapper script.
  • I have written that wrapper script ‘/usr/local/sbin/vncsession-start‘ which checks your username via the ${SUDO_USER} variable, looks up the VNC display mapping for that useraccount in ‘/etc/tigervnc/vncserver.users‘ and starts the vncsession program with the user and port as parameters.

What needs to be done?

Wrapper script:
The script ‘/usr/local/sbin/vncsession-start‘ for which I took inspiration out of the tigervnc repository looks like this:

#!/bin/bash
USERSFILE=/etc/tigervnc/vncserver.users
if [ ! -f ${USERSFILE} ]; then
echo "Users file ${USERSFILE} missing"
exit 1
fi
VNCUSER=${1:-"$SUDO_USER"}
if [ -z "${VNCUSER}" ]; then
echo "No value given for VNCUSER"
exit 1
fi
DISPLAY=$(grep "^ *:.*=${VNCUSER}" "${USERSFILE}" 2>/dev/null | head -1 | cut -d= -f1 | sed 's/^ *//g')
if [ -z "${DISPLAY}" ]; then
echo "No display configured for user ${VNCUSER} in ${USERSFILE}"
exit 1
fi
exec /usr/sbin/vncsession ${VNCUSER} ${DISPLAY}

Don’t forget to make that script executable:

# chmod +x /usr/local/sbin/vncsession-start

Sudo:
I then create a ‘sudoers‘ rule in the file ‘/etc/sudoers.d/vncsession‘ (the filename does not really matter as long as it’s in that directory and has “0440” permissions) with the following single line of content:

%vnc ALL = (root) NOPASSWD: /usr/local/sbin/vncsession-start

This sudoers rule expects that your VNC users are a member of group ‘vnc‘, so create that group and add your account(s) to it. In this example I add my own account ‘alien‘ to the new group ‘vnc‘:

# groupadd vnc
# gpasswd -a alien vnc

In order to have ‘/usr/local/sbin‘ in the $PATH when using ‘sudo‘, you must un-comment the following line in the file ‘/etc/sudoers‘ (remove the “#” at the beginning of the line):

Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

TigerVNC:
The VNC user mappings have to be added on separate lines in ‘/etc/tigervnc/vncserver.users‘. Note that each user requires a different port mapping:

:1=kenny
:2=bob
:9=alien

In order to have a sane out-of-the-box behavior for our VNC users, edit the global defaults file ‘/etc/tigervnc/vncserver-config-defaults‘ which by the way can be overruled per-user by a similarly formatted file named ‘${HOME}/.vnc/config‘ (the user would have to create it):

session=plasma
# securitytypes=vncauth,tlsvnc
# geometry=2000x1200
# localhost
# alwaysshared

The above session type “plasma” is valid because a file ‘/usr/share/xsessions/plasma.desktop‘ exists. The TigerVNC default session type “gnome” does not exist in Slackware.

Tell new VNC users to run ‘vncpasswd‘ before starting their first VNC server session, so that their sessions are at least somewhat protected.

That’s all.
Now, if you want to start a VNC server session, all you need to run as a regular user is:

$ sudo vncsession-start

and then connect to the VNC session with any ‘vncviewer‘ program. Look for my article about NoVNC if you want to give your users a web-based access to their VNC sessions. In that case you can make it mandatory for the VNC session to only bind to localhost address so that VNC sessions can not be accessed un-encrypted over the network. You can enforce this by adding a line only containing “localhost” to ‘/etc/tigervnc/vncserver-config-mandatory‘.

Good to know, it is no longer possible – but there is no longer a need – to kill the VNC server when you are done with it as was required in the past. Logging out from your graphical desktop will terminate your login session and stop the VNC server.

An additional bonus of my ‘vncsession-start‘ script is that root can run it for any user, and this makes it easy for instance to start the users’ VNC sessions in your computer’s ‘/etc/rc.d/rc.local‘ script. You just need to add the following command to start a VNC session for user ‘alien‘ – as long as a port mapping has been configured for ‘alien‘ of course:

/usr/local/sbin/vncsession-start alien

Caveat:
A TigerVNC session can only be started for/by a user if that user is not already running an interactive desktop session. This means that the root user cannot use “vncsession-start alien” to snoop my existing login session, the program will simply refuse to launch.

Thoughts?
Eric

Slackware Cloud Server Series Episode 5: Collaborative Document Editing

Hi all!
A spin-off from our previous Episode in this series is this fifth article about using Slackware as your private/personal ‘cloud server’.
Check out the list below which shows past, present and future episodes in the series, if the article has already been written you’ll be able to click on the subject.
The first episode also contains an introduction with some more detail about what you can expect.
These articles are living documents, i.e. based on readers’ feedback I may add, update or modify their content.

Collaborative document editing

In the previous Episode called “Productivity Platform” I have shown you how to setup the NextCloud platform on your Slackware server. I had promised a separate article about the addition of “collaborative editing” and this is it.

Collaborative editing on documents allows people from all over the world to have the exact same document (text, spreadsheet, presentation, vector graphics) open in a web-based online editor and collaborate on its content in real-time. Every editor can see what the others are currently working on.
The most widely used online office suite with these capabilities is Microsoft Office 365.

Of course, Microsoft 365 is not free. It uses a license model where you pay for its use per month. If you stop paying… you lose access to your online office suite and if you are unlucky, you lose access to your OneDrive files as well.

How is the situation in the Open Source world? Looking for free and open (source as well as standards-adhering!) desktop office programs, many people acknowledge that Libreoffice is an important OSOSS (Open Standards & Open Source Software) alternative to Microsoft’s Office line of programs. However… a cloud-native online web-based variant of the LibreOffice suite of programs is not trivially accessible. By nature, online software needs to be hosted somewhere and by extension, the documents you edit online need to be maintained on cloud storage as well… a challenge for free software enthusiasts when the truth is that hosting costs money. The commercially successful Microsoft Office 365 has the dominant position there.

Now, this article is going to free you (a bit) from Big Tech. I will show you how to enrich your personal Slackware Cloud Server with exactly that what seems unattainable for free software lovers: a web-based online version of LibreOffice which makes it possible for you and yours to (jointly if you want) edit the documents that you are already hosting in your NextCloud accounts.

This will be achieved by using the power of LibreOffice. An web-based version of LibreOffice exists and is called Collabora Online. It is being developed by Collabora Productivity, a subsidiary of the company Collabora who are a main contributor of LibreOffice code for many years already, and have a commercial partnership with The Document Foundation.
Formerly known under the name of LibreOffice Online, Collabora have moved the project development from LIbreOffice’s git repository to their own git repository and renamed it.
The Collabora Online Development Edition (CODE) is the version offered for free to the Open Source community (similar to the LibreOffice Community Edition) but Collabora also offer a commercial paid-for version of their Online suite. Companies using the commercial variant basically sponsor Collabora for the development of the Open Source version.

This article focuses on the integration of CODE into your NextCloud environment.

Preamble

For the sake of this article, I am using an ‘artificial’ non-existing domain name “darkstar.lan”.
I expect that your own real-life Slackware server is running Nextcloud at a publicly accessible URL, but in this article it will be:

  • https://darkstar.lan/nextcloud/

The Collabora Online Development Edition will run at its own hostname in the “darkstar.lan” domain. It is of course a virtual server running in Apache httpd, and you probably defined it in a “<VirtualHost></VirtualHost>” block. This article assumes that we are going to host CODE on a bare webserver which is already running at:

  • https://code.darkstar.lan/

Collabora Online Development Edition (CODE)

CODE is not a standalone application. It is a document-editor backend service without user interface and it communicates with your NextCloud platform via WOPI (the ‘Web Application Open Platform Interface’ protocol). It expects NextCloud to provide the user interface (a web page).
CODE is a ‘WOPI Client‘ with an API that offers editing capabilities for the documents that you host on your NextCloud (the ‘WOPI Host‘). The online editor is displayed in an embedded HTML frame right inside your familiar NextCloud environment.

The steps that we are going to take in order to integrate CODE into Nextcloud are:

  • Get a Docker container running with CODE inside and listening at 127.0.0.1:9980 for connection requests.
  • Configure Apache httpd with a reverse proxy so that this localhost port is hidden behind an externally accessible secure URL:  https://code.darkstar.lan .
  • Add the ‘richdocuments‘ app to your NextCloud server and connect it to our CODE container.
  • Tell your users!

First step: download the Collabora Online Docker image. The ‘docker run‘ command which we are going to execute in a moment would also perform that download if necessary, but it does not hurt to already make the image available locally ahead of time, so that any startup issues of the container are not clouded by connectivity issues toward the Docker Hub:

# docker pull collabora/code

The ‘docker run‘ commandline which starts the CODE container is a complex one, so let’s briefly look at its most important parameters.
As stated in the ‘preamble’, our Collabora Online Development Edition will run at hostname “code.darkstar.net” and you need to tell it explicitly which other hosts are permitted to embed it in a web-page. For us this means that our NextCloud server must be allowed to embed content originating from the CODE container when you open a document for editing. Our NextCloud hostname is passed to the container with the “aliasgroup1” environment variable.
In case you want to permit the use of your CODE Docker container to more than one Nextcloud server (for instance, you want to allow your friend who is also running a NextCloud server to use your CODE server as its collaborative editor)? Then add more hostnames using multiple ‘aliasgroupX‘ environment variables like this:

-e 'aliasgroup1=https://darkstar.lan/nextcloud' -e 'aliasgroup2=https://second.nextcloud.com'

The command to start the container is as follows:

# docker run -t -d -p 127.0.0.1:9980:9980 \
  -e "aliasgroup1=https://darkstar.lan/nextcloud" \
  -e "server_name=code.darkstar.lan" \
  -e "username=codeadmin" -e "password=My_Secret_Pwd" \
  -e "DONT_GEN_SSL_CERT" \
  -e "extra_params=--o:ssl.enable=false --o:ssl.termination=true --o:net.proto=IPv4 --o:net.post_allow.host[0]=.+ --o:storage.wopi.host[0]=.+" \
  --restart always --cap-add MKNOD collabora/code

Optionally, run with additional dictionaries.

-e 'dictionaries=nl de en fr es ..'

Through the “-p” parameter we expose the CODE container’s address at 127.0.0.1:9980 .

You see several parameters (-e "DONT_GEN_SSL_CERT" -e "extra_params=--o:ssl.enable=false --o:ssl.termination=true") that are meant to disable the container’s use of SSL encryption when communicating with the outside world. The Apache reverse proxy which we will place in front of it takes care of client-to-server encryption using a Let’s Encrypt certificate instead of the container’s self-signed certificate.

There are also a couple of parameters (-e "extra_params= --o:net.proto=IPv4 --o:net.post_allow.host[0]=.+ --o:storage.wopi.host[0]=.+") which allow the Collabora Online server to be used by hosts in your network – like your NextCloud instance. Note the use of wildcard “.+” in both the “net.post_allow.host” and “storage.wopi.host” options. I have been unsuccessful in determining a more restricted wildcard that is a better match with my network IP ranges. In the end I went for what I could find in online help forums as something which works.

The “MKNOD” capability is needed for CODE to do its work, apparently it needs to be able to create special device nodes in the filesystem using the ‘mknod‘ command. Note that according to the documentation, this makes the CODE image incompatible with Docker Swarm, but Swarm is not within the scope of my article series anyway.

If your ‘docker run‘ command-line contains a “username” and “password” parameter, then Collabora Online will enable its admin console (only accessible with the above username/password) at: https://code.darkstar.lan/browser/dist/admin/admin.html . At that URL you will find some statistics of the operations performed by the online editor.

Configuring Apache

Note that in many places on the Internet, the documentation for the reverse proxy still mentions the old LibreOffice OnLine “lool” and “loolwsd” phrases instead of using the new COllabora OnLine phrases “cool” and “coolwsd“. Luckily, Collabora’s own documentation is correct: https://sdk.collaboraonline.com/docs/installation/Proxy_settings.html. So here is what I am using on my CODE server, running on IP address:port “127.0.0.1:9980” and reverse-proxied to https://code.darkstar.lan/ .
I assume you know where to add it to your Apache httpd VirtualHost definition:

# START Reverse proxy to the CODE Docker container:
#
AllowEncodedSlashes NoDecode
ProxyPreserveHost On
ProxyTimeout 900

# Cert is issued for code.darkstar.lan and we proxy to localhost:
SSLProxyEngine On
RequestHeader set X-Forwarded-Proto "https"

# Static html, js, images, etc. served from coolwsd;
# And 'browser' is the client part of Collabora Online:
ProxyPass /browser http://127.0.0.1:9980/browser retry=0
ProxyPassReverse /browser http://127.0.0.1:9980/browser

# WOPI discovery URL:
ProxyPass /hosting/discovery http://127.0.0.1:9980/hosting/discovery retry=0
ProxyPassReverse /hosting/discovery http://127.0.0.1:9980/hosting/discovery

# Capabilities:
ProxyPass /hosting/capabilities http://127.0.0.1:9980/hosting/capabilities retry=0
ProxyPassReverse /hosting/capabilities http://127.0.0.1:9980/hosting/capabilities

# Do not forget WebSocket proxy:
RewriteEngine on
RewriteCond %{HTTP:Connection} Upgrade [NC]
RewriteCond %{HTTP:Upgrade} websocket [NC]

# Main websocket
ProxyPassMatch "/cool/(.*)/ws$" ws://127.0.0.1:9980/cool/$1/ws nocanon

# Admin Console websocket:
ProxyPass /cool/adminws ws://127.0.0.1:9980/cool/adminws

# Download as, Fullscreen presentation and Image upload operations:
ProxyPass /cool http://127.0.0.1:9980/cool
ProxyPassReverse /cool http://127.0.0.1:9980/cool
# Compatibility with integrations that use the /lool/convert-to endpoint:
ProxyPass /lool http://127.0.0.1:9980/cool
ProxyPassReverse /lool http://127.0.0.1:9980/cool
#
# END Reverse proxy to the CODE Docker container

Upgrading the Docker image for CODE

Occasionally, new versions of this docker image are released with security and feature updates. This is how you upgrade to a new version:

Grab the new docker image:

$ docker pull collabora/code

List docker images:

$ docker ps

From the output you can glean the Container ID of your CODE docker container. Stop and remove the Collabora Online docker container:

$ docker stop CONTAINER_ID
$ docker rm CONTAINER_ID

Remove the old un-used image, after looking up its ID (check the timestamp):

$ docker images | grep collabora
$ docker image rm IMAGE_ID

Start a container based on the new image, using the same ‘docker run -d‘ command as before.

CODE Docker container configuration

The default configuration for the CODE Docker image is contained in a file inside the container, called “/etc/coolwsd/coolwsd.xml” . Some of the configuration options are documented online, but the file itself is well-documented too.

After starting the container, you can copy the configuration file out of the container using docker commands as shown below. After making your changes you copy it back into the container using docker commands. When a configuration change is detected by the running container it will restart (this is why it is important to have “--restart always” as part of your ‘docker run‘ commandline).
How to do this:

Determine the CODE Docker ID by looking at the running containers:

$ docker ps | grep code

With an output like this:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e3668a79b647 collabora/code "/start-collabora-onâ?¦?" 2 weeks ago Up 7 days 127.0.0.1:9980->9980/tcp suspicious_kirch

Copy out the “coolwsd.xml” configuration file into the current directory of your host filesystem:

$ docker cp e3668a79b647:/etc/coolwsd/coolwsd.xml coolwsd.xml

Make your configuration changes to the local copy and then add it back into the container:

$ docker cp coolwsd.xml e3668a79b647:/etc/coolwsd/coolwsd.xml

The CODE container needs some time for its automatic restart, so wait a bit before you try to access it again from inside NextCloud.

Custom Docker Image

The Docker image for CODE can be customized within limits, using the variables in its configuration file, see above. But if you need substantial customization like adding more fonts or removing the 20-concurrent-open-documents limit, you can rebuild the Docker image yourself.
The code to rebuild the image can be found in the Collabora Online github repository. Use the available scripts like “install-collabora-online-ubuntu.sh” to create a custom script for your Slackware server, add your desired customizations like additional fonts or packages to that script and build your own custom Docker image with it.
Of course, the command-line to start your CODE container must be adapted to use your custom image… since you no longer want the default image to be pulled from the “collabora/code” repository in the Docker Hub.

Adding CODE to NextCloud

The preliminaries have been dealt with. Now we are getting to the core of the story: connecting your NextCloud server to the online editor backend.

The ‘richdocuments‘ app is the glue we need on NextCloud to offer collaborative document editing capabilities to our users. The human-readable name of this app as listed in your app overview is “NextCloud Office”. In fact there is an extended version of the app called ‘richdocumentscode‘ which contains an embedded version of CODE, but it is severely under-performing. I enabled the embedded CODE server in that app only to find that it crippled my complete NextCloud platform. Every action slowed down to a horrible crawl. That was the moment I decided I needed CODE to run separately, in a Docker container and outside of NextCloud.

Get the ‘richdocuments’ app by going to “Profile > Apps” and search for it in the appstore. Install it and then go to the new menu item which has suddenly appeared: “Profile > System > Administration > Collabora Online Development Server” so you can configure the URL of your Collabora Online container:

The configuration boils down to:

  • Select the radio-button ‘Use your own server‘;
  • Enter the URL “https://code.darkstar.lan/“;
  • Save

NextCloud will make an attempt to contact your CODE server and if it gets the correct responses back, the configuration page will display a green checkbox along “Collabora Online server is reachable“.

All done! We can now check out how this online office suite looks on our NextCloud.
If you are more a ‘television personality’, you can also watch a video about how to add CODE to NextCloud, created by NextCloud’s own Marketing Director (and former member of the KDE marketing team) Jos Poortvliet:

 

Using CODE in NextCloud

Once CODE is available in your NextCloud and you click on a document, you see the connection to the online editor being initialized:

This process takes a few seconds and you end up in a web-based editor which looks like:

There it is. An online web-based office suite based on LibreOffice, which is able to open and edit the documents you are storing in your NextCloud account. And if you tell NextCloud to share your document(s) with others, then they too can collaborate on your document, all together in real-time.

Thanks

You made it! After reading and implementing the ideas from these first five articles in the Series you should now have a Slackware Cloud Server up and running, full of features that everybody will be envious of, and it’s all free and under your control! No more need for Google, Microsoft, Amazon, Zoom, Dropbox and the likes. Meet and collaborate to your hearts’ desire with family, friends, co-workers or groups of Open Source developers.

Good luck and let me know about your successes, but also your frustrations and how you overcame your knowledge gaps (these articles assumed quite a bit of prior knowledge of running a webserver, configuring an online domain, managing your firewall etcetera).

There’s some episodes remaining, doing a deep-dive into Etherpad and setting the first steps into creating your own private Docker Hub without artificial usage limitations. Writing those last two episodes will take more time, so do not expect them to be released in the next couple of weeks; however I hope you will like those as well.

Eric

« Older posts

© 2024 Alien Pastures

Theme by Anders NorenUp ↑