Tag Archives: calibre

How to access your e-books online

In a previous article about the Calibre e-book library management program, I wrote that I did away with USB cable connections to transfer e-books from my Calibre library to my E-reader. Instead, I made my e-book library accessible on-line and now I am able to download my books securely over the Internet – no matter where I am, as long as I have a network connection.

In this article today, I am going to explain how I make my Calibre library accessible on-line.

Understandably, the article will be Slackware-centric, because that is where I have my Calibre library and where I run all the programs that I need. But, the technology and the configuration is generic enough that readers of this article running another distro or even employing an embedded (ARM) device will be able to make it work.

Which software is used?

Apache and PHP are standard part of Slackware Linux. I assume that you know how to setup an Apache web server. The good news is that Slackware’s default httpd configuration works out of the box. However, it does not enable a secure (https) web site. That part is on you.
If your content server is going to be used in your home network only, then HTTPS is not even needed.
I have a package for Calibre in my repository which of course you will definitely need if you want to build an e-book library.
You’ll have to download the ZIP file of the latest COPS release from https://github.com/seblucas/cops/releases – the ZIP contains the extra stuff that is not contained in the source .tar.gz file.
The remainder is just configuration.

Now let me explain my setup.

The Calibre library

It all starts with Calibre of course. It is a desktop program, available for Linux, Windows and MacOS. It stores your e-book library as a directory tree.  The directories immediately below the root of the tree are the book authors. In the root of the directory tree, Calibre creates a SQLite database file containing all the library metadata. The simple directory structure and the SQLite database file make that a Calibre library is independent of the Operating System that Calibre runs on. Its database scheme can be updated between major releases though. From within the Calibre GUI, it is however trivial to re-create this database file from scratch if it becomes corrupted.
This simple library setup has the advantage that if you store the library on a Cloud service like Dropbox, you can have access to your library from multiple computers running different Operating Systems (but preferably all the same version of Calibre). As long as one person accesses the library at any time, the Calibre database file will not get corrupted. This is how I have setup Calibre for my wife so that she can manage her e-books from every computer in the house.

Where to create the library

Myself, I use Calibre on a Linux machine which runs 24/7 (my package-build server in fact). I run a XFCE graphical desktop session inside a VNC server and that allows me to work in a desktop environment that’s always there for me. I use NoVNC to access this desktop using a web browser and that is perfect for corporate envronments which put a firewall up for everything except HTTPS traffic. Perhaps I’ll write another article sometime, explaining how to set that up.

What matters is that the Calibre library is stored on a machine with Internet access, and that this computer is running a web server like Apache. In Slackware, Apache httpd is the default webserver program.
I made this web server accessible from the Internet by configuring my ISP’s Internet router to forward ports 80 (http) and 443 (https) to the internal IP address of my Slackware machine.
For the sake of this article, let’s assume that two separate Calibre libraries exist on this computer, mine is located in /data/Calibre/EricLibrary and my wife’s books are stored in /data/Calibre/WifeLibrary . In Calibre itself, you can easily switch between libraries so that me and my wife are both able to maintain a separate collection of books. Our book tastes have no overlap…

How to configure the web server

I want both libraries to be accessible online.
I do not want to use Calibre’s own content server but instead use COPS. COPS is a collection of PHP scripts, and it does not require the Calibre program to function. It just needs access to the physical directory containing your Calibre library and it understands the Calibre SQLite database format.
If you want to get creative, you can install COPS on an Internet server – e.g. if you do not have a webserver at home. In that case, use a cloud storage service like Dropbox to keep your library in sync between your computer at home and that remote server running Apache.

My example setup is a Slackware computer at home with Apache enabled:

# chmod +x /etc/rc.d/rc.httpd
# /etc/rc.d/rc.httpd start

Apache uses the concept of a DocumentRoot, which is a directory accessible by the httpd server. Everything stored below that DirectoryRoot will be accessible via the URL of the webserver. Never store files below the DocumentRoot that you do not want to share!
I downloaded and extracted the COPS zip file directly into the Apache DocumentRoot directory (the default in Slackware is configured in the file ‘/etc/httpd/httpd.conf’ to be “/srv/httpd/htdocs/” but you can change that default location of course):

# cd /tmp
# links https://github.com/seblucas/cops/releases/download/1.1.3/cops-1.1.3.zip
# cd /srv/httpd/htdocs/
# unzip /tmp/cops-1.1.3.zip
# ln -s cops-1.1.3 mybooks
# cd mybooks
# cat <<EOT > config_local.php
$config = array();
$config['calibre_directory'] = array ("Eric" => "/data/Calibre/EricLibrary/", "Wife => "/data/Calibre/WifeLibrary/");
$config['cops_title_default'] = "Aliens Library";
$config['cops_use_url_rewriting'] = "1";
$config['cops_recentbooks_limit'] = '100';
$config['cops_update_epub-metadata'] = "1";
$config['cops_books_filter'] = array ("Books" => "!News");
EOT

The lines from the “cat << EOT” until the “EOT” on its own line is called a “here document“. You can copy/paste these lines into a terminal and it will create the file “config_local.php” with the content as shown between the “EOT” lines.
This is a configuration which allows access to two separate libraries. If you access the “recently added” section of a library it will show the latest 100 books instead of the default 20 books. It does some URL re-writing to generate cleaner URLs and it will not show any News sources (in Calibre, you can add various paid-for and free newspaper subscriptions and those will then be dpwnloaded daily, but I do not want to see those online).
Compare this configuration of mine to the ‘config_local.php.example’ file in the COPS directory. The above is not all you can configure, although it is sufficient. All configurable parameters are documented in ‘config_default.php’ which also defines their default values.

You’ll have noticed that I created a symlink to the extracted cops files, which allows me to use an easy to remember URL to access my books. Suppose my apache server is accessible as http://foo.net/ – then my COPS installation will be accessible at http://foo.net/mybooks/ .

That’s all!

Under the above ‘mybooks’ URL you will now find a COPS installation with two Calibre libraries with separate names “Eric” and “Wife”. Pick a nice template and theme, and try out the ebook-reader which is part of COPS (you  can read EPUBs in a web browser, no E-reader required).

Accessing COPS through the OPDS protocol

You want to configure your E-reader to access the library. An E-reader which contains a webbrowser is easy, just open http://foo.net/mybooks/ and click a book to download it.
Other E-readers, like FBReader for Android, are able to connect to an OPDS (Open Publication Distribution System) catalog. The COPS program exposes your library as a OPDS catalog if you just add ‘feed.php’ to the URL.
I.e. enter http://foo.net/mybooks/feed.php into the Network Library menu of FBReader and you’ll get instant access to “Alien’s Library” where you can browse Authors, Series, Categories, Recent Additions and read book descriptions, and then download the books you want to read.
If you want to always start with a list of recent library additions, you  can add ‘?page=10’ at the end of the URL so that it becomes http://foo.net/mybooks/feed.php?page=10

Hiding the cops files from the Apache DocumentRoot

Some people feel a bit anxious having these cops files accessible in their DocumentRoot – suppose someone gains access to them trough other means. You can use Apache’s “Alias” directive to install COPS in e.g. /usr/local instead. In that case, add the following lines to your /etc/httpd/httpd.conf or whatever file holds your website definition:

    <Directory "/usr/local/cops-1.1.3/">
        AllowOverride All
        Options +ExecCGI +FollowSymLinks
        Require all granted
    </Directory>
    Alias /mybooks/ /usr/local/cops-1.1.3/

You should check the validity of your httpd configuration before restarting the webserver:

# apachectl configtest
# /etc/rc.d/rc.httpd restart

And the URL to access your library will remain the same, i.e. http://foo.net/mybooks/feed.php?page=10 , only in this case your actual DocumentRoot directory will be empty.

Adding some sense of security

After following the above instructions, you now have an e-book library online whose access is not restricted in any way. If some search engine passes by it will neatly catalog all your books for other people to find. Oops!

You’ll need a basic access restriction at least. Apache offers a ‘Basic Authentication’ setup which will prompt anyone who tries to access your library for a valid account and password.
To achieve this, let’s first create a password file containing the account/password combinations for the people you want to grant access. For the sake of this example, I will grant myself and my wife access by creating account/password combinations for us in a file which we will then use in  our Apache configuration.
The first ‘htpasswd’ command below has an additional ‘-c’ parameter to create the file “/etc/httpd/passwords/htaccess.opds”:

# mkdir -p /etc/httpd/passwords/
# htpasswd -b -c /etc/httpd/passwords/htaccess.opds eric ericspassword
# htpasswd -b /etc/httpd/passwords/htaccess.opds wife wifespassword

The file will then have these contents, you see that the passwords are now MD5 encrypted:

eric:$apr1$OtKDA27W$l2ac4DAhGCG53igy6jT5A/
wife:$apr1$Zql/HEUC$wrNckoe57YPC0u2w8mL/M0

Read “man htpasswd” to find out more about this command and its parameters.

Next you need to change the Apache <Directory></Directory> block which I provided above. It will become like this:

    <Directory "/usr/local/cops-1.1.3/">
        AllowOverride All
        Options +ExecCGI +FollowSymLinks
        AuthBasicAuthoritative off
        AuthUserFile /etc/httpd/passwords/htaccess.opds
        AuthType Basic
        AuthName "OPDS Server"
        Require valid-user
        # Require all granted
    </Directory>
    Alias /mybooks/ /usr/local/cops-1.1.3/

Once you made these changes, validate the configuration and restart the webserver:

# apachectl configtest
# /etc/rc.d/rc.httpd restart

Now you will be greeted by an authentication request, next time you access your library. Only those users whose username/password combinations are stored in “/etc/httpd/passwords/htaccess.opds” will be able to get access.

Using encrypted HTTP

You should realize that these passwords will still be transmitted in cleartext if your webserver is not using HTTPS. It is possible for people to sniff the network connection and find your credentials.

I think however that instructions about enabling HTTPS for your Apache belongs in another blog post. Let me know in the comments section below if you have a need for such an article.
Also, let me know if parts of the above instructions are too cryptic and I will update the text where needed.

Good luck! Eric

Calibre 4.2.0 for Slackware with no external dependencies

One of the programs I use a lot on all my computers is Calibre, the E-book library management system. My wife and I both have Kobo e-readers, and on my Android phone I use FBReader to read my books on the road whenever I have some free time.

Having Calibre on a computer still requires a USB cable to transfer e-books from the computer to your ereader/phone, but networked alternatives exist. Calibre comes with a content server which can make your e-book library accessible online (on your LAN or on the Internet). Earlier versions of the Calibre content server were too resource-intensive and therefore I have chosen another solution for online access to my books. That’s COPS. which is short for “Calibre OPDS PHP Server“. OPDS (Open Publication Distribution System) Is the protocol through which E-readers can access online libraries. COPS allows me to download new books to read from my Calibre library over the wireless network to both my Kobo e-reader and to FBReader on my phone. No more cables needed!

FBReader on Android

FBReader accesses e-book library via COPS

Calibre is of course more than a library manager and a content server – the software comes bundled with its own e-book reader, a format converter, and an e-book editor!
Calibre is written in Python. For its graphical interface it heavily uses PyQt5 which in turn is based on Qt5 libraries.
I have been offering calibre packages in my repository for over eight years, starting with the 0.7.20 release as far as I could see (then still Qt4 based). This roughly coincided with my wife buying her first E-reader. The program has evolved a lot over the years and is much better than the proprietary library management software that ships with the major E-reader hardware.

Kovid Goyal is the primary developer, and he released a new major version of Calibre a few weeks ago. The new 4.x series delivers a completely re-written reader, a new content server with basically the same management and editing capabilities as the calibre desktop program, and a stark increase in amount and version numbers of the dependent software. For instance, Calibre 4 now depends on features in Qt 5.13.0 and PyQt 5.13.0. Several new pieces of sofware are listed in the dependencies list now, which were not required for Calibre 3.

The challenge was to create a solid Slackware package for Calibre 4. I know, you can simply download pre-built binaries for a generic Linux platform, and SlackBuilds.org even offers a build script to package these binaries for you, but it is much more fun, and very educational, to compile all of it yourself. In addition you’ll get binaries which are native to your Slackware distro instead of having to resort to binaries that were compiled on Ubuntu or Debian and do unpeakable things to your system when you’re not looking.

Luckily I have taken a week off from work, so I had time to spend on a rewrite of calibre.SlackBuild.

I also decided to do away with external dependencies. Until now, you would have to install podofo and unrar in order to fully use Calibre’s functionality. WIth my new calibre-4 package all dependencies are bundled inside. This includes Python 2.7.17 and Qt 5.13.1 as well as podofo and unrar libraries. This bundling makes it possible to build a fully functional Calibre on Slackware 14.2 (I had to add a newer version of the hunspell spell-checker because Slackware 14.2 has a version that is too old).

You can still compile a calibre package without all these internal libraries. For instance if you already have Qt 5.13.1 installed you can reduce the size of the package significantly by recompiling it. The calibre.SlackBuild will find your Qt5 libraries and won’t compile an internal version then. Same for Python 2.7. But I would recommend building the package always with the python interpreter and the slew of python modules included. It does not take that much time and you’ll be independent of updates in Slackware. Also, I do not offer packages for all those required Python modules anyway.

Here’s a couple of example command lines in case you would want to recompile. If you want create a package without any external dependency:

# BUILD_QT=YES BUILD_PYTHON=YES BUILD_MTP=YES  ./calibre.SlackBuild

In case you already have Qt 5.13.0 or newer installed and want to shrink the size of the package:

# BUILD_PYTHON=YES ./calibre.SlackBuild

Note:
On Slackware 14.2 you will have to install meson, python3 and python3-setuptools from my repository or else the compilation will fail.  Slackware-current contains all of them already.

Note:
If you have issues with Calibre 4 and think you can convince me to re-add a Calibre 3 package to my repository, let me know in the comments section below.

Note:
If you use XFCE or some other desktop environment which does not support StatusNotifier (and only supports the old X11 based XEmbed specification like the systray), then Calibre will not be able to display its system tray icon. You will have to install ‘xfce4-statusnotifier-plugin’ and its dependency ‘libdbusmenu-gtk’. Both are available as packages in my repository. After installing, right-click on the XFCE status bar, select ‘Panel > Add new items’ and add ‘Status Notifier Plugin’.

Get the new ‘calibre-4.2.0‘ package from my repository (or any mirror) and enjoy your e-book library!

If you want to know how to properly setup COPS and serve your e-book library on the internet (with or without authentication) let me know and I will devote an article to the topic. Perhaps in more generic terms, I should address the topic of reverse proxying.

Eric

Calibre 3.30.0 for Slackware with internal Qt5 libraries

It took me quite a while to release a new package for Calibre, the e-book library manager. That had a reason.

In July I switched the Qt5 package in my repositories to version 5.11 to support the latest KDE Plasma5 software and because it offers advantages over the previous 5.9 releases. Unfortunately, as I found out soon afterwards, the Calibre software fails to work with Qt 5.11 – its GUI components were not built and there was no obvious error to explain why.

Therefore I had to re-visit the calibre.SlackBuild‘s internals and try to revive the internal functions that compile an embedded Qt library set. This was last tested in the early days of my Calibre packages when Qt4 was the running champion. Adding internal Qt5 support was quite a different beast. Qt5 is a lot bigger than the venerable Qt4 so the build process needed some pruning to keep the compilation times acceptable and the package size under control.

That took a full week’s nights of compiling, debugging, recompiling and so on… hence the lack of updates on the ‘ktown’ front where I should perhaps pay some attention to a recent poppler update in slackware-current. But I managed to add Qt 5.9 internal library support to the calibre package.
My package for Slackware 14.2 still depends on an external ‘qt5’ package (to keep the package size small and because calibre works just fine with the qt 5.7 which is available from my own repository and the SBo script repository). The package for Slackware -current on the other hand was built with an embedded Qt 5.9, which means that its external dependency list shrunk to just ‘podofo’ and ‘unrar’.

Grab the new ‘calibre’ package and enjoy your e-book library!

Almost weekend again – what’s in store

Just a quick recap of my work during the past week (well… the work that I do besides my paid-for work):

I updated my packages for calibre and chromium with new versions. I updated the set of “compat32” packages for a multilib setup on slackware64-current to match the Slackware packages contained in the new Slackware 14.2 Beta 2.

And I updated the “plasma” package set of my KDE5 (aka Plasma 5) repository on ‘ktown‘; this is also just also for slackware-current. Plasma was upgraded to 5.5.4 which is a new bugfix release.

And there is a bit more, still in the pipeline. I have stamped a “version 0.5.0” onto my liveslak scripts and I am currently in the process of generating new ISO images for my Slackware Live Edition (in full Slackware, Plasma5, Mate and slimmed-down XFCE variants).

After I upload the new ISOs I will update the git repository with liveslak-0.5.0 sources. More about that hopefully tomorrow if my testing yielded good results.

Cheers, Eric

LibreOffice 5.0.2 and Calibre 2.39.0 packaged

libreoffce_logoThe Document Foundation announced version 5.0.2 of their free Office Suite a few days ago. LibreOffice 5.0.2 is the second update to the “5” major release. Again this is a bugfix release for Linux, no new functionality has been added. According to the Document Foundation “LibreOffice 5.0.2 is targeted to technology enthusiasts, early adopters and power users. For more conservative users, and for enterprise deployments, TDF suggests the “still” version: LibreOffice 4.4.5“.

The new LibreOffice 5.0.2 packages have been compiled for users of Slackware-current only – after all, you are the power users of Slackware. I offer 4.4.5 packages for Slackware 14.1 where a wee bit more conservatism is a good thing. The 4.4.5 packages should also work on -current, but I have not tested that.

lo502_about

There was a bug report in the comments section of my previous LibreOffice blog post: audio and video files embedded in presentations suffer from delays when opened. This appears to happen in all builds of LibreOffice that use gstreamer-1.x (like my LibreOffice 5 packages) whereas the bug does not appear when LibreOffice has been configured to use gstreamer-0.x (like my LibreOffice 4 packages). The bug has been reported over a year ago, but it does not show a lot of movement.

I decided to stick with gstreamer-1.x, to see if the new release is still affected. Let me know! If the bug still shows, I will compile LO 5.0.3 against gstreamer-0.x again.

For download locations, see below.

calibreicoAlso, I released packages for the latest version of Calibre. I quit following the weekly Calibre update cycle, and at some point noticed that the Calibre developer himself also switched from weekly updates to bi-weeklies.

Calibre 2.x uses Qt5 for its GUI so you’ll have to install a couple of dependencies as well: qt5 of course, and podofo. The remainder of the dependencies (several python libraries) has been built into the package so that they do not have to be installed separately. Available for both Slackware 14.1 and Slackware-current, you can grab the calibre package off  the Slackware server or any other mirror that carries my repository:

Cheers! Eric