Main menu:

Sponsoring

Please consider a small donation:

 

Also appreciated: support me by clicking the ads (costs nothing) :-)

 

Or you can donate bitcoin:

 

Thanks to TekLinks in Birmingham, AL, for providing colocation and bandwidth.

Page Rank

Fame

FOSS Force Best Blog--2013 Award

Recent posts

Recent comments

About this blog

I am Eric Hameleers, and this is where I think out loud.
More about me.

Search

My Favourites

Slackware

Calendar

August 2014
M T W T F S S
« Jul    
 123
45678910
11121314151617
18192021222324
25262728293031

RSS Alien's Slackware packages

RSS Alien's unofficial KDE Slackware packages

RSS Alien's multilib packages

Meta

Setting up your own Mozilla Sync server

Last sunday was not a good weekend, from a software usability point of view.

I had not worked with the laptop (running 64-bit Slackware-current) for a few days and thought I’d do some preparation work for the upcoming KDE release (you know, checking out if any dependencies needed adding or updating, build script checks and such). However, I spent most of my time swearing at the Firefox browser.

What happened? I still do not know the cause, but within seconds of starting Firefox, it would consume roughly 70% of all available memory…  that is 2 GB of real RAM and 3 GB of swap. Even with one tab open, and a simple HTML page loaded, and with all my add-ons disabled, the situation did not improve.

Let me tell you, in such a situation a man can go nuts! Every click on a Firefox menu or button, or a page flip, would cause the laptop to freeze for up to 20 seconds while Linux was swapping massive amounts of useless data back and forth between RAM and swap.

Finally and enraged, I killed the firefox process, deleted my ~/.mozilla/firefox directory (to be honest, I renamed it) and fired up the browser again. And voilà! Firefox was behaving like a gentleman again. Of course, that left me without my bookmarks, browsing history, browser preferences, and saved passwords. Or did it?

A month or so ago, I decided to try the feature which is supported in Firefox since version 4, called “Mozilla Sync”. This feature allows you to store your bookmarks, browsing history, browser preferences, and saved passwords on a remote server. It even allows you to work on multiple computers and have the same Firefox configuration on all these computers. That’s a pretty cool feature. I have setup sync on my Android phone’s Firefox browser too. Having enabled the sync feature was my life saver during this weekend horror show! After I had removed the ~/.mozilla/firefox directory I quickly configured Firefox with my sync key and moments later I had my old browser configuration back… sans the ludicrous memory consumption. I was a happy puppy again.

That could have been the end of my blog post.

However I wrote this as a preamble to what I am going to show you next – how to setup your own Sync Server at home (or at work).

When I was contemplating the (dis)advantages of using Mozilla Sync, I was not too fond of the idea that the Sync Server which Firefox will use by default is hosted by Mozilla. Even though your data is supposedly being encrypted with your unique sync key before it gets uploaded to Mozilla, it feels awkward to see all your passwords and browsing data being transfered to a 3rd party. Furthermore, the geek in me was just dying to setup a sync server of my own.

Luckily, Mozilla is open as always and makes the Open Source Sync Server code available to everyone who wants to play with it. And setting it up proved to be not difficult at all. Note: Mozilla have re-designed and re-written the original Sync Server (it was called “Mozilla Weave Server” at the time, and was almost impossible to setup… I tried, back in october 2010). It’s now written in Python, and the steps to create your own Sync Server are described on this page: http://docs.services.mozilla.com/howtos/run-sync.html. That page is what you should follow. The rest of my article tries to bring more detail to the process of configuring your Slackware box to become a real Sync Server.

Of course  you want to install this on a Slackware server! There are some explicit dependencies as mentioned on the Mozilla page I referred to earlier, but python 2.6 and sqlite3 are already suppied by Slackware, and I created a package for virtualenv (for Slackware 13.1 and 13.37) which you can download here: http://slackware.com/~alien/slackbuilds/virtualenv/ .

Note (20120627): it looks like installing this virtualenv package is no longer needed, but I did not check this. A comment from Old_Fogie at the bottom of this package indicates that virtualenv is downloaded automatically during the “make build” you will run as part of the next step in my procedure. My suggestion is to skip the installation of my virtualenv package at first, and see what happens you run the “make build”. If you get an error about missing virtualenv, then install that package and re-run “make build”…

The Sync Server code is hosted in a Mercurial repository and since Slackware ships a Mercurial client, that code is easy to check out to a local directory where you continue to build the server binary:

$ mkdir -p /usr/local/server-full
$ cd /usr/local/server-full
$ hg clone https://hg.mozilla.org/services/server-full/ .
$ make build

Now, this server is ready to run, but unless you make some configuration changes it is not going to do you much good.

By default, the server stores your sync data in a sqlite database, which is fine with me (you can use a real MySQL server database if you want), but it will create the sqlite database in /tmp which is entirely inappropriate. Further down I will show you what to modify so that your clients’ sync data go somewhere less “volatile”.

The python-based Sync Server can run all by itself on a configurable port number (5000 by default) which is just fine if you are its only user in your little LAN. But even Mozilla advises to put a real webserver between you and the Sync Server. For this I used Slackware’s apache webserver. This means, you will need one additional package, mod_wsgi which you can download here: http://slackware.com/~alien/slackbuilds/mod_wsgi/ . I created a separate system user and group (weave:sync) which is going to run the Sync Server so that it can not interfere with Apache if anything goes crazy:

# groupadd sync
# useradd -g sync weave

If you omitted the creation of the homedirectory for this “weave” user account, then create it like this:

# mkdir /home/weave
# chown weave:sync /home/weave
# chmod 711 /home/weave

You then create a new file “/etc/httpd/extra/httpd-wsgi.conf” with the following content (adapt to your actual sync server hostname):

   <Directory /usr/local/server-full>
      Order deny,allow
      Allow from all
    </Directory>

    <VirtualHost *:80>
      ServerName sync.myserver.net
      DocumentRoot /usr/local/server-full
      WSGIProcessGroup sync
      WSGIDaemonProcess sync user=weave group=sync processes=2 threads=25
      WSGIPassAuthorization On
      WSGIScriptAlias / /usr/local/server-full/sync.wsgi
      CustomLog /var/log/httpd/sync.myserver.net-access.log combined
      ErrorLog  /var/log/httpd/sync.myserver.net-error.log
    </VirtualHost>

As you may notice, this tells that a script called “sync.wsgi” is going to be spawned as user “weave:sync” whenever a client connects to http://sync.myserver.net . You will have to modify your apache main configuration too (i.e. “/etc/httpd/httpd.conf“), in order to make it load the mod_wsgi module and use its configuration file. Add these lines to the httpd.conf:

# Activate the WSGI module (serving python binaries to the web):
Include /etc/httpd/extra/mod_wsgi.conf
# Act as a reverse proxy to the Mozilla Sync server:
Include /etc/httpd/extra/httpd-wsgi.conf

… followed by an Apache restart (you can run “apachectl configtest” first, to check for mistakes in the modified configuration if you can’t affort your webserver to go down thanks to a typo)

Now comes the part where you configure the Sync Server configuration files. Take a look inside your directory “/usr/local/server-full” where the Sync Server code has been downloaded and built. You will find the aforementioned “sync.wsgi” script there. Open the file in an editor. Read through the file and change it to something which you find better suited. For instance, I changed “/tmp” to “/var/tmp” in the line “os.environ['PYTHON_EGG_CACHE'] = ‘/tmp/python-eggs’”:

os.environ['PYTHON_EGG_CACHE'] = '/var/tmp/python-eggs'

Next, edit the file “development.ini” (that name is configured in “sync.wsgi” in case you want to give it another). In that file, I changed the location of another log file so that it gets written in “/var/tmp”. The line “args = (‘/tmp/sync-error.log’,)” changes to:

args = ('/var/tmp/sync-error.log',)

This “development.ini” file also mentions the filename of the actual sync configuration (yeah, my idea too was that this level of nesting configuration files seems unwanted): it has the name “etc/sync.conf” relative  to the server’s directory. It requires some editing! For instance, look for the two lines that look like this and change the path name to a safer location in your filesystem! I decided to create the SQL database file in the “weave” user’s own homedirectory, so I changed the line “sqlite:////tmp/test.db” to:

sqluri = sqlite:////home/weave/my_weave_server.db

The default value for the quota assigned to each sync account is too low (it’s set to 5 MB). Mozilla states that 99.9% of all clients which connect to its public server use less than 25 MB of quota, so use that value. The line “quota_size = 5120″ changes to:

quota_size = 25600

Then, look for the following line and make absolutely certain that it is changed to the Apache http:// URL of your Sync Server, i.e. the URL that the sync clients are going to use:

fallback_node = http://localhost:5000/

You can play with other options in this file to see what they can offer. For instance,  I disabled “captcha” and did not enable SSL, but you may decide that this is important to you.

Don’t forget to restart your apache server if you have not yet done so.

And now it is time to tell your Firefox browser to use this new service!

  • Select “Tools > Setup sync…”:

  • In case this is the first time you talk to your Sync Server, you need to setup an account, so click the button “Create a New Account“. This opens a new dialog box where you can type your email address (this is your account name), associated password and then in the dropdown do not select “Firefox Sync Server“… instead, you choose “Use a custom server“, it allows you to enter the URL for your brand-new private Sync Server. Once the account has been created, Firefox will show you the sync key it will be using to encrypt your data. That key is important, you will need it if you want to add more computers to your sync account. And that’s basically it – the sync process will start immediately:

  • Else if you already have a Sync Account and want to add a second computer to it, you can click on “Connect” in the first dialog which will open a new dialog. There are two ways to add this computer to your existing sync account. Either enter the code which is being shown on another computer which is already syncing to the server; or else you click on “I don’t have the device with me“. This is what you would usually do I guess… it is combersome to search for another computer or hand-held if all you need to do is know your sync key:

  • If you know your sync key you can enter that in the following dialog, along with your account details and the server URL:

  • The sync key which is associated with your sync account can be retrieved at any moment should you have forgotten to write it down. Just go to a computer which has been setup for Sync, and open your Firefox browser’s Preferences at the “Sync” tab. Under “Manage your account” you will find a menu item to display your key (as well as a way to change your account’s password). More help can be found at http://support.mozilla.com/kb/what-firefox-sync .

Have fun! Eric

Comments

Comment from Ellendhel
Posted: September 1, 2011 at 17:51

Very interesting post, I will probably try to install this service soon.

Thanks a lot Eric!

Comment from eternauta2001
Posted: September 1, 2011 at 22:25

Nice! Thanks for the builds.

Comment from escaflown
Posted: September 2, 2011 at 00:58

Thanks for libreoffice, Eric.

Comment from MartinM
Posted: September 11, 2011 at 23:22

Nice job man!
Keep it UP!

Comment from Tyler
Posted: January 2, 2012 at 10:41

Alas, I can’t seem to get mine working :(
I keep getting error messages about “‘sync.wsgi’ cannot be loaded as Python module.”

[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] mod_wsgi (pid=4902): Target WSGI script ‘/var/mozilla-sync/server-full/sync.wsgi’ cannot be loaded as Python module.
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] mod_wsgi (pid=4902): Exception occurred processing WSGI script ‘/var/mozilla-sync/server-full/sync.wsgi’.
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] Traceback (most recent call last):
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/sync.wsgi”, line 74, in
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] application = loadapp(‘config:%s’% ini_file)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 203, in loadapp
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return loadobj(APP, uri, name=name, **kw)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 224, in loadobj
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return context.create()
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 617, in create
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return self.object_type.invoke(self)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 109, in invoke
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return fix_call(context.object, context.global_conf, **context.local_conf)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/util/fixtypeerror.py”, line 57, in fix_call
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] val = callable(*args, **kw)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/deps/server-core/services/baseapp.py”, line 342, in make_app
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] app = klass(urls, controllers, params, auth_class)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/deps/server-storage/syncstorage/wsgiapp.py”, line 102, in __init__
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] auth_class)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/deps/server-core/services/baseapp.py”, line 125, in __init__
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] signal.signal(signal.SIGTERM, self._sigterm)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] ValueError: signal only works in main thread

Comment from alienbob
Posted: January 2, 2012 at 10:59

Looking at the path names (/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/) this seems to be a pre-packaged mozilla-sync server for some other distribution than Slackware. If that is the case, you should talk to the one who created the package.
What I used was the code available at https://hg.mozilla.org/services/server-full and the Python 2.6 which is part of Slackware. No issues there.

Eric

Comment from old fogie
Posted: June 27, 2012 at 05:53

Hello Eric !

Thank you for this guide. I just ran thru this tonight and have the following comments:

There appears to be an error in this howto at:

Quote: You then create a new file “/etc/httpd/extra/httpd_wsgi.conf” with the followin….

Because later in the guide we are instructed to edit httpd.conf with:

Quote: Include /etc/httpd/extra/httpd-wsgi.conf

The error is in the “_” and it should be the same as what we put in the httpd.conf file, that is, it should be a “-” , eg: “httpd-wsgi.conf”

So the line in the howto article should read:

You then create a new file “/etc/httpd/extra/httpd-wsgi.conf” with the followin….

Regarding dependencies, the make process downloaded the following files, note the version of virtualenv is greater than the version of the package we are advised to install. Which is of no surprise as quite some time has passed since this article was written. So I’m not sure if we need to install the virtualenv package anymore ?? as the make process will download as needed.

Here is an abbreviated version of the output from make process:

Downloading distribute-0.6.27.tar.gz
Downloading MoPyTools-3.2.tar.gz
Downloading Paste-1.7.5.1.tar.gz
Downloading PasteScript-1.7.5.tar.gz
Downloading PasteDeploy-1.5.0.tar.gz
Downloading flake8-1.3.1.tar.gz
Downloading Distutils2-1.0a3.tar.gz
Downloading virtualenv-1.7.2.tar.gz
Downloading pypi2rpm-0.6.3.tar.gz
Downloading argparse-1.2.1.tar.gz
Downloading PasteDeploy-1.5.0.tar.gz
Downloading nose-1.1.2.tar.gz
Downloading WebTest-1.3.4.zip
Downloading WebOb-1.2.zip

Lastly, I suggest appending to the article for those, that are following your recommendations to the letter:

mkdir /home/weave
chown /home/weave weave:sync
touch /home/weave/test.db
chown weave:sync /home/weave/test.db

Perhaps we should chmod 0700 /home/weave as well ? I haven’t tested that yet.

Thanks again Eric ! Great job!

Comment from alienbob
Posted: June 27, 2012 at 15:32

Hi Old Fogie,

Those are all valid and useful comments, thank you! I have made sure to implement them into the main article, and made some other enhancements as well.

Just hours ago, I installed the new native Firefox for Android on my smartphone and used my sync server to be up and running very quickly. The server is a big time saver.

Cheers, Eric

Comment from krizzo
Posted: July 16, 2012 at 17:01

I ran into an issue where I get “Already in Use”. This after I deleted the DB file which I confirmed with sync devs that’s the only location the account name is stored.

I was wondering if you had this issue and what would be causing it?

Comment from Janis
Posted: September 1, 2013 at 09:40

hmm.. when I try to enter my brand new sync server url (as sync.domain.lv, with or without http://) into respective seamonkey registration dialog, I see “please enter valid server URL”.
Is there anything to do to get around?
Janis

Comment from Janis
Posted: September 1, 2013 at 09:45

sorry, as usually with me – solved by myself after asking a question (problem lied with permissions to log file)

Comment from Janis
Posted: September 1, 2013 at 14:11

The only thing remains unclear – if I add a device to the personal sync server’s account – how browser deciphers all the information necessary for connection using just 12 alpha-numerical characters? Or some (a lot of ) information goes to Mozilla anyway?

Comment from sethuper
Posted: October 14, 2013 at 23:41

I have followed step by step in this tutorial, everything is describe, how to install and configure own Firefox Sync Server (Weave) with MySQL.

Comment from Janis
Posted: October 15, 2013 at 07:50

slackware’ish startup script would be very welcome

Pingback from Alien Pastures » Chromium for Slackware, with salt & pepper
Posted: November 23, 2013 at 00:35

[…] so I can keep them synchronized across computers – just like Firefox for which I setup my own private Sync server, However, Chrome is a closed-source app and I prefer to run open source self-built stuff where […]

Comment from alienbob
Posted: November 25, 2013 at 12:47

Hi Janis, a startup script would not be needed because the sync server is configured to be part of your Apache service.

Eric

Comment from Janis
Posted: November 25, 2013 at 14:17

Hi, Eric!
It seems it’s not my case – I did (I think so) as you write, but anyway I have to run sync server binary to get synced

Comment from alienbob
Posted: November 25, 2013 at 14:52

Hi Janis

The point is that the definition line “WSGIScriptAlias / /usr/local/server-full/sync.wsgi” in the “/etc/httpd/extra/httpd-wsgi.conf” file triggers the start of the sync server when a client connects.

Eric

Comment from Janis
Posted: December 6, 2013 at 09:29

Hi!
I re-checked – i have the definition, but still have to run bin/paster by hand. Respective the http server logs are empty

Comment from alienbob
Posted: December 6, 2013 at 11:26

Might be interestng to find out if this is a Apache 2.2 versus 2.4 difference. I run this on Slackware 13.37 using apache 2.2, while Slackware 14.1 has Apache 2.4.

Eric

Comment from Janis
Posted: December 14, 2013 at 09:40

yes, I have 2.4.3 installed on that server

Comment from krizzo
Posted: December 14, 2013 at 17:04

Janis There is a slight change with Apache 2.4 that you need to make in the configurations you can see the changes in the slack docs wiki.

Comment from alienbob
Posted: December 14, 2013 at 17:29

FYI that SlackDocs location is http://docs.slackware.com/howtos:network_services:weave#configuration

Eric

Comment from Janis
Posted: December 14, 2013 at 21:44

I have a config like seen in wiki from the very beginning. Despite that the service is not spawned on reqest for sync and there are NO error messages anywhere – virtual server logs are empty, sync-error.log is not even created.
As I wrote before – only bin/paster helps.

Comment from Janis
Posted: December 14, 2013 at 21:48

May be it doesn’t like the fact I use mysql as DB?

Comment from Janis
Posted: December 14, 2013 at 22:03

or some over-restrictive apache cfg:
ServerLimit 6
StartServers 2
MaxClients 20
MinSpareThreads 10
MaxSpareThreads 20
ThreadsPerChild 10
MaxRequestsPerChild 700
KeepAlive On
MaxKeepAliveRequests 200
KeepAliveTimeout 5
?

Write a comment