Hi all!
This is the second episode in a series of articles I am writing about using Slackware as your private/personal ‘cloud server’ while we are waiting for the release of Slackware 15.0.
Below is a list of past, present and future episodes in the series. If the article has already been written you’ll be able to access it by clicking on its subject.
The first episode also contains an introduction with some more detail about what you can expect from these articles.
- Episode 1: Managing your Docker Infrastructure
- Episode 2 (this article): Identity and Access management (IAM)
Setting up Keycloak for Identity and Access Management to provide people with a single user account for all the Slackware Server Services we will be creating.- Preamble
- Setting up Keycloak
- The MariaDB database
- Network defaults in Docker
- Private network for Keycloak
- MariaDB access from the network
- Keycloak container
- MariaDB considerations during Keycloak setup
- The admin user
- Reverse proxy setup for keycloak
- Initial configuration of Keycloak
- Considerations
- Keycloak discovery
- Done
- Thanks
- Attribution
- Episode 3: Video Conferencing
- Episode 4: Productivity Platform
- Episode 5: Collaborative document editing
- Episode 6: Etherpad with Whiteboard
- Episode 7: Decentralized Social Media
- Episode 8: Media streaming platform
- Episode 9: Cloudsync for 2FA Authenticator
- Episode X: Docker Registry
Identity and Access Management (IAM)
When you run a server that offers all kinds of web-based services, and you want all these services to be protected with an authentication and authorization layer (i.e. people have to login first and then you decide what kind of stuff they can access) it makes sense to let people have only one identity (one set of credentials) that can be used everywhere. Not exactly ‘Single Sign On‘ because you may have to logon to the various parts separately but you would be using that single identity everywhere.
Slackware comes with Kerberos and OpenLDAP servers which can be used together with for instance Samba to create a Single Sign On environment on a local network. But since I am looking for something more infrastructure agnostic which works all across the Internet, I ended up with something I vaguely knew since it’s used in places in our own company to provide credential management: Keycloak.
Keycloak offers web-based Open Source Identity and Access Management (IAM). Using Keycloak, I will show you how to add authentication and authorization to the applications that I will be adding to my “Slackware Cloud Server”.
Keycloak can be used to manage your users, but if you already manage your users in an OpenLDAP server or in Active Directory, Keycloak can use these as its back-end instead of its own local SQL database.
The users of our Slackware server will authenticate with Keycloak rather than with the individual applications. Ideally, once your users have logged into Keycloak they won’t have to login again to access a different application. Likewise, Keycloak provides single-sign out, which means after a user logs out of Keycloak, that will apply to all the applications that use Keycloak.
Keycloak can also act as an Identity Broker for “social login” so that you are able to use social networks like Facebook, Google etc as Identity Providers. Our Slackware server applications that ask Keycloak to handle the user login and authorizations don’t know and don’t care about exactly how the authentication took place, as long as it’s a process the administrator trusts.
For the purposes of this article however, I will limit the use of Keycloak to authentication through OpenID Connect (OIDC) or SAML 2.0 Identity Providers. The applications I will discuss in the upcoming articles (NextCloud, Jitsi Meet, Quay, Docker Registry) all support the OIDC protocol for off-loading authentication / authorization so we’ve got that covered.
Keycloak does not only manage the identities that allow your users to login; Keycloak can also manage authorizations.
You may want to allow specific users a different level of access to your applications, or prevent access to some of them entirely. You can assign roles to users or add them to groups and then configure the access to your applications using the available roles or group memberships.
Preamble
For the sake of this instruction and for future articles in this series, “https://sso.darkstar.lan/auth” is the base URL that eventually the Keycloak application will be available at at.
Configuring the DNS for your own domain (will probably be something else than “darkstar.lan”…) with new hostnames and then setting up web servers for the hosts in that domain is an exercise left to the reader.
Before continuing, please ensure that your real-life equivalent for the following host has a web server running:
- sso.darkstar.lan
It doesn’t yet have to serve any content yet but its URL needs to be accessible to all applications and users that you want to give access to your Slackware Cloud Server. We will add some blocks of configuration to the VirtualHost definition during the steps outlined in the remainder of this article.
Using a Let’s Encrypt SSL certificate to provide encrypted connections (HTTPS) to your webserver is documented in an earlier blog article.
Setting up Keycloak
NOTE!
Starting with Keycloak 20 (released in November 2022), the WildFly based distribution is no longer supported. For the newer Quarkus distribution of Keycloak, check out the new documentation .
This article targets a pre-20 release Docker container. I will have to update the text to make it reflect the new Docker options.
Working with a Docker based server infrastructure is something I covered in the previous article of this series. I assume you have Docker up and running and are at least somewhat comfortable creating containers and managing their life-cycle.
Keycloak can be installed on a bare metal server but I have opted to go for their Docker-based install instead. The configuration will be stored in a MariaDB backend and an Apache reverse proxy will be the frontend which will handle the incoming connection requests and also enforces data encryption using a Let’s Encrypt SSL certificate.
The Keycloak github contains an example of a docker-compose solution with MariaDB and Keycloak in two separate containers (see https://github.com/keycloak/keycloak-containers/tree/master/docker-compose-examples) however the documentation states that this does not work currently because MariaDB does not start fast enough for Keycloak.
Also I think that having your database inside the container, or even mounting the host’s “/var/lib/mysql” directory into the container, will make backups difficult. So, our Keycloak application will connect to Slackware’s own included MariaDB database via the host’s IP address. There’s a paragraph further down where I share my considerations about this setup.
Here’s what we are going to do now: create the MariaDB database, create a private network for the Keycloak container so that it at least has some isolation from other containers, ensure that we have some kind of DNS service for the custom network (this example uses dnsmasq for that) and finally: start the Keycloak service as a single container, no Compose needed.
The MariaDB database
MariaDB in Slackware 15.0 is at version 10.5.13. Considering future upgrades: any security update in a stable Slackware release should not introduce breaking changes, but it’s always good to have read the MariaDB pages on database upgrades to avoid surprises. And always make backups that you have tested (are the backups useable).
Login to MariaDB as the administrator (commonly the user ‘root’):
$ mysql -uroot -p
Create the database:
> CREATE DATABASE IF NOT EXISTS keycloakdb CHARACTER SET utf8 COLLATE utf8_unicode_ci;
Create a database user for Keycloak:
> CREATE USER 'keycloakadm'@localhost IDENTIFIED BY 'your_secret_passwd';
Grant all privileges:
> GRANT ALL PRIVILEGES ON keycloakdb.* TO 'keycloakadm'@localhost;
Reload the grant tables (flush privileges) and quit the management interface:
> FLUSH PRIVILEGES; > quit;
Network defaults in Docker
Docker claims a network range when it starts, in order to connect its containers to each other and to the real world. By default, the range is 172.17.0.0/255.255.0.0 and there is no reason to change that unless it collides with pre-existing network segments in your LAN. In that case, you would want to edit “/etc/docker/daemon.json” and add something like this to define a custom Bridge IP address and container IP range:
{
"bip": "172.100.0.1/16",
"fixed-cidr": "172.100.0.0/24",
}If you would want your containers to be in yet another IP range, you could add something like:
{
"default-address-pools": [
{
"base": "192.168.56.0/21",
"size": 28
}
]
}
Docker needs a restart to pick up the changes. Newly created networks will then be dynamically assigned a /28 subnet of the larger IP range “192.168.56.0/21” unless you manually specify other ranges (see below).
Private network for Keycloak
We create a Docker network “keycloak0.lan” just for Keycloak, and make it use a different IP range than Docker’s default. Otherwise it would share its network with the rest of the containers and we want to create some level of real containment, as well as the ability to assign the Keycloak container a fixed IP address.
We need a fixed IP address in order to assign a hostname on the host so that Sendmail allows emails to be sent from within the container. Otherwise we would get ‘Relaying denied: ip name lookup failed” error from Sendmail.
Summarizing: the “keycloak0.lan” network will have a range of 172.19.0.0/16; and the fixed IP address for the Keycloak service will be 172.19.0.2 (since 172.19.0.1 will be the IP address of the Docker bridge).
$ docker network create --driver=bridge --subnet=172.19.0.0/16 --ip-range=172.19.0.0/25 --gateway=172.19.0.1 keycloak0.lan
Add host and network to /etc/hosts and /etc/networks:
$ grep keycloak /etc/hosts
172.19.0.2 keycloak keycloak.keycloak0.lan
$ grep keycloak /etc/networks
keycloak0.lan 172.19
My assumption is that your host does not act as the LAN’s DNS server. We will use dnsmasq to serve our new entries from /etc/hosts and /etc/networks – dnsmasq will be our local nameserver. For this we use the default, unchanged “/etc/dnsmasq.conf” configuration file.
You will also have to add this single line at the top of “/etc/resolv.conf” first, so that all DNS queries will go to our local dnsmasq:
nameserver 127.0.0.1
If you have not yet done so, (as root) make “/etc/rc.d/rc.dnsmasq” executable and start dnsmasq manually (Slackware will take care of starting it on every subsequent reboot):
# chmod +x /etc/rc.d/rc.dnsmasq
# /etc/rc.d/rc.dnsmasq start
Make Sendmail aware that the Keycloak container is a known local host by adding a line to “/etc/mail/local-host-names” and restarting the sendmail daemon:
$ grep keycloak /etc/mail/local-host-names
keycloak.keycloak0.lan
If you use Postfix instead of Sendmail, perhaps this is not even an issue, but since I do not use Postfix I cannot tell you. Leave your comments below if I should update this part of the article.
MariaDB access from the network
Since our Keycloak container will connect to the MariaDB server over the network, we need to grant the Keycloak database account access to the database when it logs in from the network instead of via the localhost.
Login to ‘mysql’ as the admin (root) user and execute these commands that come on top of the ones that we already executed earlier:
> CREATE USER 'keycloakadm'@'%' IDENTIFIED BY 'your_secret_passwd';
> GRANT ALL PRIVILEGES ON keycloakdb.* TO 'keycloakadm'@'%';
> FLUSH PRIVILEGES;
> quit;
Keycloak container
We use a Docker container to run this IAM service. Remember that containers do not persist their data. In our case it means that the complete configuration which is needed by the Keycloak application inside that container in order to start up properly, has to be passed as command-line parameters to the ‘docker run‘ command.
This is how we start the Keycloak container, connecting it to the newly created network, assigning a static IP, using the ‘keycloak.lan‘ network gateway as the IP address for the MariaDB and passing it the required database properties.
By default, Keycloak listens at port “8080” but that is a portnumber which is (ab)used by many applications including proxy servers. Instead of accepting the default “8080” value we do a port-mapping and make Keycloak available at port “8400” instead by using the “-p” argument to ‘docker run‘.
We also pass the front-end URL which is going to be used by every application that wants to interact with the service (https://sso.darkstar.lan/auth). This URL is actually served by a Apache httpd reverse-proxy which we will put between the exposed port of the Keycloak container and the rest of the world:
$ mkdir -p /usr/share/docker/data/keycloak
$ cd /usr/share/docker/data/keycloak
$ echo 'keycloakadm' > keycloak.dbuser
$ echo 'your_secret_passwd' > keycloak.dbpassword
$ docker run -d --restart always -p 8400:8080 --name keycloak \
--net keycloak0.lan \
--network-alias keycloak.keycloak0.lan \
--ip 172.19.0.2 \
-e DB_VENDOR=mariadb \
-e DB_ADDR=172.19.0.1 \
-e DB_DATABASE=keycloakdb \
-e DB_USER_FILE=$(pwd)/keycloak.dbuser \
-e DB_PASSWORD_FILE=$(pwd)/keycloak.dbpassword \
-e KEYCLOAK_FRONTEND_URL=https://sso.darkstar.lan/auth \
-e PROXY_ADDRESS_FORWARDING=true \
jboss/keycloak \
-Dkeycloak.profile.feature.docker=enabledThat last argument ‘-Dkeycloak.profile.feature.docker=enabled‘ enables the “docker-v2” protocol in Keycloak for creating a Client in the realm. A Keycloak ‘Client ID‘ is what we need to connect an application to Keycloak as the login provider. This is a topic we will visit in future Episodes.
This ‘docker-v2‘ protocol is not fully compatible with the default OIDC protocol ‘openid-connect‘ and while not enabled in Keycloak by default, it is an officially supported Client Protocol.
We need ‘docker-v2‘ when we want the Docker Registry to be able to use Keycloak for authentication / authorization (the topic of Episode 6 in our Series).
A note about the Keycloak service URL (
https://sso.darkstar.lan/auth). In this example I use a hostname “sso.darkstar.lan” but you will have to pick a name which fits your own network. This hostname will be used a lot and visible everywhere, so if you are paranoid and want to avoid people realizing that your “sso.” host is the key to all user-credentials, you might want to chose a more inconspicuous name like “pasture.darkstar.lan“. YMMV.
Also, the path component “/auth” in the URL is fixed and immutable, unless you want to hurt yourself. I tried to change that path component to something that made more sense to me but that “/auth” is so ubiquitous as hard-coded strings throughout the code that I quickly gave up.
Remember that our Keycloak container listens at the non-default TCP port 8400 on the docker0 interface, which we will use for the reverse proxy setup later on.
MariaDB considerations during Keycloak setup
At first, I tried with mounting the host’s MariaDB server socket in the Keycloak container and accessing the database through it via the ‘localhost’ target like below:
-e DB_ADDR=localhost -v /var/run/mysql/mysql.sock:/var/run/mysql/mysql.sock
…but JBoss would not be able to connect for whatever reason, and the resulting error was ‘connection refused‘.
I have tried various locations for the UNIX socket inside the container:
/var/run/mysql/mysql.sock
/var/run/mysqld/mysql.sock
/var/run/mysqld/mysqld.sock
/tmp/mysql.sock
… but none of those worked either.
Eventually I had to make MariaDB listen on all interfaces and rely on my firewall to block external access, for instance using:
# iptables -A INPUT -s 172.19.0.2 -p tcp --destination-port 3306 -j ACCEPT
This is the relevant MariaDB configuration line:
$ grep bind-address /etc/my.cnf.d/server.cnf
bind-address=0.0.0.0
If you have a better solution let me know! Of course, the easy way out of this dilemma is to deploy Keycloak using Docker Compose and give it its own internal database, but we need to be 100% sure that the SQL database server is up & running before we start the Keycloak container as part of the Docker infrastructure in “/etc/rc.d/rc.local“.
Also, my goal was to have a single SQL server on my host that would be the backend database for every application that needs one. A lot easier to backup and restore.
The admin user
Create the admin user once the container is running or else you won’t be able to connect at all (use ‘docker ps’ to find the containerID), and restart the container after that, but WAIT AFTER THAT RESTART LONG ENOUGH FOR THE INITIAL CONFIGURATION TO FINISH… or you’ll end up with Keycloak trying to initialize MariaDB twice, resulting in errors and failure.
This is how you run the command to create the admin user on the host, to be executed inside the running container:
$ containerID=$(docker ps -qaf "name=^keycloak$")
$ docker exec <containerID> /opt/jboss/keycloak/bin/add-user-keycloak.sh -u admin -p your_secret_admin_pwd
$ docker restart <containerID>
Write that password down in a safe place and remove that command-line from your Bash history! It is the key to the Realm. Literally.
Reverse proxy setup for keycloak
In your Apache configuration for the “sso.darkstar.lan” host you need to add these lines to create a reverse proxy that connects the client users of your Keycloak service to the service endpoint:
SSLProxyEngine On
SSLProxyCheckPeerCN on
SSLProxyCheckPeerExpire on
RequestHeader set X-Forwarded-Proto: "https"
RequestHeader set X-Forwarded-Port: "443"
<LocationMatch /auth>
AllowOverride None
Require all granted
Order allow,deny
Allow from all
</LocationMatch>
ProxyPreserveHost On
ProxyRequests Off
ProxyVia on
ProxyAddHeaders On
ProxyPass /auth http://127.0.0.1:8400/auth
ProxyPassReverse /auth http://127.0.0.1:8400/authRestart Apache httpd after making this change.
Initial configuration of Keycloak
Now that Keycloak is up and running behind a reverse proxy and interfaces with the world via the URL https://sso.darkstar.lan/auth , and has an admin account, we are going to create a “realm“. A realm is a space where the administrator manages identifiable objects, like users, applications, roles, and groups. A user belongs to only one realm and will always login to that realm.
Our realm will be named “foundation“. Here we go:
- Point your browser at https://sso.darkstar.lan/auth/
- Add a new Realm:
- Click to open Admin Console (https://sso.darkstar.lan/auth/admin).
- Logon with above configured ‘admin’ credentials.
- Hover the mouse over the ‘Master‘ dropdown in the top-left corner.
- Click ‘Add realm‘, enter Name: “
foundation“. - Click ‘Create‘, add a Display name “
My Cloud Foundation” (pick any name you like), click ‘Save‘.
- (Optionally) configure email via SMTP:
- Click ‘Email‘ (top of the page).
- Enter your LAN’s SMTP server, the SMTP port (587 if you enable StartTLS at the bottom), and a reasonable ‘From‘ email address.
- (Optionally) configure other parameters specific to your setup.
- Click ‘Test Connection‘.
- If your Keycloak container could successfully connect to the SMTP server, click ‘Save‘ and now Keycloak will be able to send you relevant emails, provided that you configured a working email address for the admin user of course.
- Add a first user:
- Click ‘Users‘ (left-hand menu) > ‘Add user’ (top-right corner of table)
- Add user “
alien” with full-name “Alien BOB” (of course, use whatever makes sense for you).
- Set initial password to be able to login:
- Click ‘Credentials‘ (top of the page).
- Fill in ‘Set Password‘ form with a password like “WelcomeBOB!”.
- (Optionally) click ‘ON‘ next to ‘Temporary‘ so that it toggles to ‘OFF‘. This removes the requirement for the user to update password on first login.
- Click ‘Set password‘.
- Test the new user login in a different browser (or logoff the admin user first):
- Open User Console: https://sso.darkstar.lan/auth/realms/foundation/account
- Login as ‘
alien‘ and you’ll see the message: “You need to change your password to activate your account.“ - Set a new (permanent) password and explore the configurable user profile settings, for instance enabling 2-Factor Authentication (2FA) using an app like Authy or Google Authenticator.
Considerations
Your keycloak is now running at https://sso.darkstar.lan/auth and the user login-page for the “Foundation” realm is at https://sso.darkstar.lan/auth/realms/foundation/account . That URL is nowhere to be found easily, so if you don’t know it is there (or if you don’t know about Keycloak’s URL path template) then users may complain about not finding it.
On the one hand, this adds some “security through obscurity” if you’re the paranoid type.
Plus, Keycloak will redirect users to the correct login page anyway when an application asks for a credential check.
On the other hand, you want to give your server’s users a nice experience.
As a courtesy to your users, you could consider adding a HTML redirect to the root of that server. Assuming that https://sso.darkstar.lan/ just serves an empty page or spits out a “page not found” error, you could paste the following HTML snippet into an ‘index.html‘ file in the DocumentRoot directory, which will cause an immediate redirect from the URL root to the user login page for the “foundation” realm:
<html> <head> <title>Slackware Cloud Server Logon</title> <meta http-equiv="refresh" content="0; URL="https://sso.darkstar.lan/auth/realms/foundation/account"> </head> <body bgcolor="#ffffff"></body> </html>
Keycloak discovery
In the previous section I mentioned that it is not trivial for users to discover the URL of their Keycloak profile page. But how do applications do this when they want to interact with Keycloak?
OpenID Connect Discovery is a layer on top of OAuth 2.0 protocol which allows a client application not only to authenticate a user but also to obtain certain user characteristics (called “claims” in OAuth2.0) like full name or email address. To obtain these claims, the client application connects to a well-known URL where it can find out which claims are supported.
Keycloak supports this OpenID Connect Discovery and provides a Discovery URL at “/auth/realms/<realm>/.well-known/openid-configuration”. For our Keycloak server, that will be:
https://sso.darkstar.lan/auth/realms/foundation/.well-known/openid-configuration
You can query this URL for instance via the ‘curl’ program and if you also have the ‘jq‘ program installed from my repository you can generate nicely formatted human-readable colorized output:
$ curl https://sso.darkstar.lan/auth/realms/foundation/.well-known/openid-configuration | jq
The output of that command is lengthy but it starts with:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5749 100 5749 0 0 41485 0 --:--:-- --:--:-- --:--:-- 41659
{
"issuer": "https://sso.darkstar.lan/auth/realms/foundation",
"authorization_endpoint": "https://sso.darkstar.lan/auth/realms/foundation/protocol/openid-connect/auth",
"token_endpoint": "https://sso.darkstar.lan/auth/realms/foundation/protocol/openid-connect/token",
"introspection_endpoint": "https://sso.darkstar.lan/auth/realms/foundation/protocol/openid-connect/token/introspect",
"userinfo_endpoint": "https://sso.darkstar.lan/auth/realms/foundation/protocol/openid-connect/userinfo",
"end_session_endpoint": "https://sso.darkstar.lan/auth/realms/foundation/protocol/openid-connect/logout",
"frontchannel_logout_session_supported": true, foundation
"frontchannel_logout_supported": true,
"jwks_uri": "https://sso.darkstar.lan/auth/realms/foundation/protocol/openid-connect/certs",
"check_session_iframe": "https://sso.darkstar.lan/auth/realms/foundation/protocol/openid-connect/login-status-iframe.html"
,
"grant_types_supported": [
........
We will be able to use this Discovery protocol when setting up an Etherpad container in Episode 6.
Done
We are now ready to use Keycloak as IAM service for other applications that are waiting for us to install and configure. This setup is secure by default, but I do invite you to read more about the advanced configuration of Keycloak, they have extensive documentation at https://www.keycloak.org/documentation . For instance, do you want new users to be able to register themselves? Allow them to configure 2-Factor Authentication? Update their profile? Force password expiry? Custom password complexity configuration?
You may want to add groups to the realm and not just users, and limit the use of certain applications to specific groups (you may not want to let everyone use the Jitsi video conferencing platform for instance).
If your network already manages its users in an Identity provider like LDAP or Kerberos, Active Directory or eDirectory, or if you want to allow people to use their Google, Facebook etc identities to authenticate against your Keycloak, you should look into the functionality behind ‘Identity Providers’ in the left sidebar:
In that case, Keycloak will not store user identity information in its MariaDB database but instead use these Identity Providers as the remote backend.
Thanks
I hope you made it this far… and you liked it. Leave your comments, encouragements, fixes and general feedback in the section below.
Cheers, Eric
Attribution
Keycloak architecture images have been taken from the keycloak.org/documentation web site. The remainder are screenshots taken from my own Keycloak instance.
Great!
Thanks!
I have added a section about “Keycloak Discovery” which I will refer to in Episode 6 (Etherpad).
Great job ! thanks a lot !
there’s a typo in ‘ docker run keycloak ‘ , miss anti-slash after ‘–net keycloak0.lan’
Indeed mab974, well spotted and thanks for reporting. I have fixed it.
Hey, I’ve found another typo: On creating the dbuser file, you wrote keykloak.dbuser instead of keycloack.dbuser.
I have run into a snag getting keycloak going. I have created the keycloak database and user and saw no errors. After a lot of research, I have made no headway in solving this. I deleted the database and started over thinking I had a typo or some mistake. I did follow the first article and got portainer-ce going, so all previous steps in the prior article were done and worked. I cut and pasted Eric’s commands and adjusted as needed (passwords, server domain/TLD).
“docker ps” gives the below
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ddfe34sg4 jboss/keycloak “/opt/jboss/tools/do…” 2 weeks ago Restarting (1) 6 seconds ago keycloak
It appears to restart over and over and never get going. I have stopped it then started it, same thing
“docker logs “2ddfe34sg4” has the below:
/opt/jboss/tools/docker-entrypoint.sh: line 20: /usr/share/docker/data/keycloak/keycloak.dbuser: No such file or directory
-Dkeycloak.profile.feature.docker=enabled
“/usr/share/docker/data/keycloak/keycloak.dbuser” does have the correct user in it. Line 20 of docker-entrypoint.sh doesn’t appear to be germain to “keycloak.dbuser”, though I could be wrong.
Any help would be greatly appreciated. I have spent quite a bit of time trouble shooting but can’t find any hint of a solution or what I may have done wrong. I’m hoping someone here has done this and may see something obvious.
AlienBOB, thank you so much for your time creating packages and very well written, informative posts. I have used Slackware since version 10.2 and your packages are a tremendous addition: EFG for IPTables, LibreOffice, Wine, Steam,and Chromium-ungoogled in particular.
Can you post the full commandline you are using to start that container? I think you may be omitting something. Try putting everything on one line, without backslashes to wrap multiple lines.
alienbob,
Thank you for your reply. Below is the command. I did take your command in the above guide and paste it into kwrite, adjusted as necessary then copied that to paste on the command line the first time through two weeks ago. Just now, I did it all on one line as suggested. I have been using “docker stop ” and “start” to stop and start and restart it. When I re-ran the original command, it complained that that containerID was in use, so I “docker rm ” then re-ran the command. It then went into the restarting loop again. It just occurred to me I don’t have a “/var/www/htdocs/auth” directory yet. Although, that doesn’t seem to me to be the nature of the error.
docker run -d –restart always -p 8400:8080 –name keycloak –net keycloak0.lan –network-alias keycloak.keycloak0.lan –ip 172.19.0.2 -e DB_VENDOR=mariadb -e DB_ADDR=172.19.0.1 -e DB_DATABASE=keycloakdb -e DB_USER_FILE=$(pwd)/keycloak.dbuser -e DB_PASSWORD_FILE=$(pwd)/keycloak.dbpassword -e KEYCLOAK_FRONTEND_URL=https://sso.folklandmanagement.com/auth -e PROXY_ADDRESS_FORWARDING=true jboss/keycloak -Dkeycloak.profile.feature.docker=enabled
Is that file “/usr/share/docker/data/keycloak/keycloak.dbuser” perhaps only readable by root? That would explain why you get an error about the file while it exists.
Alternatively, try specifying the user and password using DB_USER and DB_PASSWORD variables on the command-line instead of DB_USER_FILE and DB_PASSWORD_FILE.
Like: “-e DB_USER=your_db_user -e DB_PASSWORD=your_db_password”
“/usr/share/docker/data/keycloak/keycloak.dbuser” is read and write by root and read by the group and user, which are both root. I think that may be a problem. I did as you said specifying the user and password directly in the command line to bypass the file(s) above that and it looked like it worked: “docker ps” and it had up time, no more “restarting.” Once I went into portainer to look at stats, it said it was restarting, so I did “docker ps” and it said it was. “docker logs ” is very long but it does appear to be the same 30 or so line that repeat. Keycloak appears to stay up for about 10 seconds the goes into restarting loop. New issue but clear progress in the right direction. Does the server need to have access to the internet for keycloak to start and stay up? I am scanning through the log and it appears as though it is looking for internet but I am not certain on that. The error appears to be the same thing repeating many times. There is more to it than what I have below, but I think this is the relevant section.
02:06:14,355 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.controller.AbstractAddStepHandler$1@4ba5c3d3 for operation add at address [(“interface” => “management”)] failed — java.util.concurrent.RejectedExecutionException
02:06:14,355 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.controller.AbstractAddStepHandler$1@5d127c61 for operation add at address [
(“core-service” => “management”),
(“management-interface” => “http-interface”)
] failed — java.util.concurrent.RejectedExecutionException: java.util.concurrent.RejectedExecutionException
This server is not connected to the internet right now, I have it prepared and set up (with your EFG) to be our gateway, firewall, DHCP server at the office. ETH1 is 192.168.1.1; the same as my router here at the house. I think that is causing a problem here since the home router has that same address. It’s not really a problem because it all works fine at the office and it didn’t seem worth it to me to change it, on occasion I have taken it to the office to work on it there. I don’t have time during the day to set this server up at the office so I mostly have it at the house and I work on it at night. Tomorrow, I will take it to the office and see if an internet connection solves this. There may be something you recognize in the above error that makes more since to you than it does me. Some googling makes it look like a local DNS error but some other points to a java issue. I see some “blocking timeout” with regard to the database import (but we are not importing a database). See below. The error there looks the same as mine.
https://serviceorientedarchitect.com/wflyctl0348-timeoutexception-while-running-keycloak-in-a-docker-container-with-an-external-database-mariadb/
Thank you again for all of your help.
The domain you are using in the front-end URL (sso.folklandmanagement.com) should exist and it should be associated with that keycloak server you want to start. I see that the DNS entry exists, but when Keycloak starts, will it be able to listen at https://sso.folklandmanagement.com/auth ?
But for sure, you have an IP conflict when you configured that host to have an IP address which is already claimed by your LAN gateway. First fix that by assigning a different IP address for the host’s eth1 interface.
Your connection issue may also be the result of a firewall configuration that is too restrictive. The EFG script knows only lo, eth0, eth1 etc interfaces but if you run Docker, you will see a lot of other interfaces as well: the docker0 interface and lots of br-* interfaces. If you do not deal with those in the EFG script, their traffic will probably be blocked.
I just posted (or though I did) a reply. It was long. I forgot to “reply” to your post and ended up creating a new post in the root of the comments. That was about 15 minutes ago and not sure it went through. If not, I will address rc.firewall first.
# Internet Interface
INET_IFACE=”eth0″
INET_ADDRESS=”192.168.127.145″
#INET_ADDRESS=”66.180.233.60″
# Local Interface Information
LOCAL_IFACE=”eth1″
LOCAL_IP=”192.168.1.1″
LOCAL_NET=”192.168.1.0/24″
LOCAL_BCAST=”192.168.1.255″
DOCK_IFACE=”docker0″
BR_IFACE=”br-+”
# Localhost Interface
LO_IFACE=”lo”
LO_IP=”127.0.0.1″
Then farther down:
#Docker
$IPT -A INPUT -p ALL -i $DOCK_IFACE -j ACCEPT
$IPT -A INPUT -p ALL -i $BR_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $DOCK_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $BR_IFACE -j ACCEPT
Lastly, “/var/www/htdocs/auth” was created. I can’t find any info on the correct owner and permissions. Right now, it is:
drwxr-xr-x 2 root root 4096 Oct 31 13:25 auth/
That seems wrong to me.
Thanks again!!
It is not needed, in fact I think it can be detrimental, to create a “/var/www/htdocs/auth” directory. The /auth part of the Keycloak URL is handled by Keycloak itself, through the reverse proxy setup. Hence also the “
< LocationMatch /auth>< /LocationMatch>” directive that controls access to that URL, instead of using something like “< Directory /auth>”Also, from the “INET_ADDRESS=”192.168.127.145?” line I assume that your Keycloak server is not directly connected to the Internet but hidden behind another router. Carefully determine where you actually need to use the real Internet IP address instead of this LAN IP address.
alienbob,
I removed “/var/www/htdocs/auth.” You are correct that right now this server is not connected directly to the internet but behind a router (Slackware 14.0 server). Once the setup is complete, it will be connected directly and the 66.180.233.60 will be uncommented and the 192.168.127.145 will be commented out in rc.intet1.conf.
As alienbob mentioned at the very beginning, it does look like a firewall issue. I had left out the dnsmasq part. When I started rc.dnsmasq, it complained that port 53 was in use. “chmod -x rc.bind” and “sh rc.bind stop” then allowed rc.dnsmasq to start fine. Once I did that, there were some obvious firewall issues/complaints in the logs/terminal output. That was yesterday, today I can not reproduce those errors but docker logs does have a lot “connections refused” so I still think the firewall is the root problem. I have not solved the issue yet I think my next step is digging into rc.firewall. It appears 172.19.0.2 (MariaDB) is not causing problems, rc.firewall does have:
#MariaDB listen on all interfaces; Keycloak network
$IPT -A INPUT -s 172.19.0.2 -p tcp –destination-port 3306 -j ACCEPT
“172.19.0.1” is full of connection errors in the docker log.
As a test, I added:
IPT -A INPUT -s 172.19.0.1 -p tcp –destination-port 8080 -j ACCEPT
$IPT -A INPUT -s 172.19.0.1 -p tcp –destination-port 8400 -j ACCEPT
to rc.firewall to see the impact, but now when starting the keycloak container I get:
Error response from daemon: driver failed programming external connectivity on endpoint keycloak (2fdbf21447b3a1d4d8a1f8c3abbde0bf9acfeb55fbeda6f90d0e8ad1e54409f3): (iptables failed: iptables –wait -t nat -A DOCKER -p tcp -d 0/0 –dport 8400 -j DNAT –to-destination 172.19.0.2:8080 ! -i br-d63410966225: iptables: No chain/target/match by that name.
(exit status 1)).
Anyway, looks like I am headed in the right direction and will post again once solved.
I failed to mention, I do have the below in rc.firewall, but it is not enough:
#Docker
$IPT -A INPUT -p ALL -i $DOCK_IFACE -j ACCEPT
$IPT -A INPUT -p ALL -i $BR_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $DOCK_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $BR_IFACE -j ACCEPT
At the very top
DOCK_IP=”172.19.0.2″
is specified.
I have not made any progress. I think the road block found in the logs is:
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to address=(host=172.19.0.1)(port=3306)(type=master) : Socket fail to connect to host:172.19.0.1, port:3306. Connection refused (Connection refused)
Caused by: java.sql.SQLNonTransientConnectionException: Socket fail to connect to host:172.19.0.1, port:3306. Connection refused (Connection refused)
From alienbob’s reply dtd October 27, 2022 at 15:03, the firewall script from EFT does (I manually added it) specify the docker IP address of 172.19.0.2 and the bridge of 172.19.0.1 and then as stated in my above post
#Docker
$IPT -A INPUT -p ALL -i $DOCK_IFACE -j ACCEPT
$IPT -A INPUT -p ALL -i $BR_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $DOCK_IFACE -j ACCEPT
$IPT -A OUTPUT -p ALL -o $BR_IFACE -j ACCEPT
Should the docker address in rc.firewall be 172.17.0.1? “Network Defaults in Docker” section here says this is the default address (range) and it is what ifconfig has. I did test this and did change it in rc.firewall to 172.17.0.1 and restarted docker but no change in behavior. Is the above log errors the database refusing the connection (eg: wrong password) or is it the firewall not allowing it? From what I am grasping, MariaDB listens on all interfaces; as configured in the instructions above and has no network of its own. That being said, the connection to the bridge is being refused and may not have anything to do with MariaDB, correct? Searching the internet about error, the closest I found suggested the database wasn’t up and running yet. Logging in shows it has been up 42 minutes. Ifconfig has the docker bridge at 172.19.0.1 and docker as 172.17.01. Based on the command to start the keycloak container, (only two of the lines),
–ip 172.19.0.2 \
-e DB_ADDR=172.19.0.1 \
I don’t have anything at 172.19.0.2, I assume this is the network address keycloak is creating, correct (and not docker itself)? When the container is not running, there would be nothing there, if I understand correctly. Although, when I start keycloak, ifconfig doesn’t have anything there. Is that because of the connection refused on 172.19.0.1? Once this is up, 172.19.0.2 comes up?
I appreciate anyone’s help, the deeper I dig and try to understand, the more confused I get. I don’t understand how alienbob can accomplish what he does after work and in his off-time. After a full day of work for me then working on this in the evenings, my brain in mush and I end up going in circles. His much better grasp of all this is probably a big help.
I do not think that the error “Socket fail to connect to host:172.19.0.1, port:3306. Connection refused (Connection refused)” is firewall-related.
It is the MariaDB TCP port, it receives a connection request and apparently the database server refuses your connection.
Check the permissions for the database user and make sure the user’s permissions include connecting remotely instead of just via localhost.
alienbob,
Thank you for your reply. I checked the permissions and keycloakadm had ‘%’ for connection. Figuring I messed something up weeks ago in the creation of the database and user to begin with, I did remove the database and user and stared over, redoing all of the MariaDB parts. With that done, I still got the “Connection refused.”
Checking the access again:
MariaDB [(none)]> SHOW GRANTS FOR ‘keycloakadm’@’%’;
+————————————————————————————————————+
| Grants for keycloakadm@% |
+————————————————————————————————————+
| GRANT USAGE ON *.* TO `keycloakadm`@`%` IDENTIFIED BY PASSWORD ‘*898’ |
| GRANT ALL PRIVILEGES ON `keycloakdb`.* TO `keycloakadm`@`%` |
+————————————————————————————————————+
Lastly, I specifically added access for 172.19.0.1 with:
GRANT ALL ON keycloakdb.* to ‘keycloakadm’@’172.19.0.1’ IDENTIFIED BY ‘mYvErySekretpassword’ WITH GRANT OPTION;
Flushed permissions, quit, logged back in, then:
MariaDB [(none)]> SHOW GRANTS FOR ‘keycloakadm’@’172.19.0.1’;
+———————————————————————————————————————+
| Grants for keycloakadm@172.19.0.1 |
+———————————————————————————————————————+
| GRANT USAGE ON *.* TO `keycloakadm`@`172.19.0.1` IDENTIFIED BY PASSWORD ‘*5523’ |
| GRANT ALL PRIVILEGES ON `keycloakdb`.* TO `keycloakadm`@`172.19.0.1` WITH GRANT OPTION |
+———————————————————————————————————————+
Am in interpreting this wrong? Looks like it should work to me. The MariaDB error log has no mention of any connection attempts.
On the command line I can log in:
#mysql -u keycloakadm -pMyhighlysekretpw
then:
MariaDB [(none)]> SELECT USER(),CURRENT_USER();
+———————–+———————–+
| USER() | CURRENT_USER() |
+———————–+———————–+
| keycloakadm@localhost | keycloakadm@localhost |
+———————–+———————–+
It shows me logged in from localhost. Is there a way to log in from the docker bridge IP? I can ping that ip, so it is functional. It seems this would yield useful messages. I did build and install your liveslack docker image. Once started I figured if I tried to log into the database there it may use the bridge but it looks like MariaDB isn’t in there. Thanks again for your help!!
Instead of running “mysql” from the host command-line, you should first start a shell inside the container. /bin/bash”. Inside the container you can play around with connection tests towards the MariaDB database. The database server will indeed appear to be located at the gateway IP of the Docker network, not at localhost. Are you able to access the SQL port (3306) from within the container?
You can do that with the command “docker exec -it
I always test connectivity with the command telnet:
root@marco:/etc# /etc/rc.d/rc.cups start
cups: started scheduler. [ OK ]
root@marco:/etc# telnet localhost 631
Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.
^]
telnet> q
Connection closed.
root@PartedMagic:/etc# /etc/rc.d/rc.cups stop
cups: stopped scheduler. [ OK ]
root@PartedMagic:/etc# telnet localhost 631
Trying 127.0.0.1…
telnet: connect to address 127.0.0.1: Connection refused
Maybe a litle bit old fashioned, but for me it works.
Also, if you try to connect to the MariaDB database and get an error message, stop the MariaDB service on the machine you’re trying to connect to, and if you then get a different error message with the same connection attempt, it is probably not firewall related, right?
alienbob and Marco, thank you for your replies. (Edit: I got it fixed, I will leave the below in case someone else has the same problem I did.)
Marco, very good idea. I did as you suggested and stopped rc.mysql and got the same result when trying to connect from alienbob’s slacktest container. I also stopped rc.firewall and tried to connect; same result whether started or stopped. From the slacktest command prompt, I can ping the bridge at 172.19.0.1. From the terminal in the container:
bash-5.1$ mysql -u keycloakadm -h 172.17.0.1 -p
Enter password:
ERROR 2002 (HY000): Can’t connect to MySQL server on ‘172.17.0.1’ (115)
When I try to connect to host 192.168.1.1 (eth1 of the bare metal, which does answer pings from within slacktest) from within the container, I get the same message. On the bare metal, I get the same “ERROR 2002” if I specify “-h 192.168.1.1” or any IP address. If I use “-h localhost” it connects fine. It seems like MariaDB is not listening on all interfaces, but “server.cnf” does have “bind-address=0.0.0.0”
“netstat -tulpn | grep 3306” results in no output.
MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE ‘PORT’;
+—————+——-+
| Variable_name | Value |
+—————+——-+
| port | 0 |
+—————+——-+
That doesn’t look right, I assume it should say 3306 rather than 0.
All this makes me think there is an issue with MariaDB an it is not listening at all and only responds to localhost connection attempts. The Slackware64 15.0 machine I am sitting at has the same behavior, so does the slacktest container. I am thinking I missed something major in my MariaDB setup. “netstat -tlnp” nothing is listening on port 3306.
On the machine I am sitting at (my desktop, not the server I am configuring), I did follow “https://docs.slackware.com/howtos:databases:install_mariadb_on_slackware” for the setup of MariaDB but still get the same result. From what I have read, you can specify the port in server.cnf, but shouldn’t be necessary as it would use the default port. I did add the port and “skip-networking=off” (unnecessary, I think the default is off) to the config file and restart, no luck. I suspect it is something very simple to get it to listen on that port, but everything I have found reveals no reconfiguration that I can find. “netstat -ln | grep mysqld” is blank as well. It is clear the issue is MariaDB not listening on port 3306 but rather 0.
Oh…you have to edit rc.mysqld and comment out SKIP=”–skip-networking” Adding that to the server.cnf will not do it, rc.mysql apparently takes priority. And now….FROM SLACKTEST I CAN NOW LOG INTO THE DATABASE AT 172.19.0.1!!!!! And the machine I am sitting at. I did have to restart docker, but keycloak starts and runs!!!!!! Wow, I’m pretty excited, I have been fighting this thing for a while. It is too late now to finish the setup, but I can move on to the rest of the configuration. Thanks again for all the help Eric and Marco!!!
Hello. GBranco again!
Is it possible to upgrade keycloak without fier of braking the all server login? Can you walk me through?
If you need functional support, my advice would be to open a request on the Keycloak community web site: https://www.keycloak.org/community
There’s a warning that I added to the beginning of this article, informing about incompatible changes between Keycloak versions before and after the 20 release. I have not attempted an upgrade myself, simply started from scratch because I don’t have that big an amount of users.
I still need to update this article with installation instructions that apply to the new Keycloak versions. I have written all of that down in my personal log, just not yet into this blog.