Last sunday was not a good weekend, from a software usability point of view.
I had not worked with the laptop (running 64-bit Slackware-current) for a few days and thought I’d do some preparation work for the upcoming KDE release (you know, checking out if any dependencies needed adding or updating, build script checks and such). However, I spent most of my time swearing at the Firefox browser.
What happened? I still do not know the cause, but within seconds of starting Firefox, it would consume roughly 70% of all available memory… that is 2 GB of real RAM and 3 GB of swap. Even with one tab open, and a simple HTML page loaded, and with all my add-ons disabled, the situation did not improve.
Let me tell you, in such a situation a man can go nuts! Every click on a Firefox menu or button, or a page flip, would cause the laptop to freeze for up to 20 seconds while Linux was swapping massive amounts of useless data back and forth between RAM and swap.
Finally and enraged, I killed the firefox process, deleted my ~/.mozilla/firefox directory (to be honest, I renamed it) and fired up the browser again. And voilà! Firefox was behaving like a gentleman again. Of course, that left me without my bookmarks, browsing history, browser preferences, and saved passwords. Or did it?
A month or so ago, I decided to try the feature which is supported in Firefox since version 4, called “Mozilla Sync”. This feature allows you to store your bookmarks, browsing history, browser preferences, and saved passwords on a remote server. It even allows you to work on multiple computers and have the same Firefox configuration on all these computers. That’s a pretty cool feature. I have setup sync on my Android phone’s Firefox browser too. Having enabled the sync feature was my life saver during this weekend horror show! After I had removed the ~/.mozilla/firefox directory I quickly configured Firefox with my sync key and moments later I had my old browser configuration back… sans the ludicrous memory consumption. I was a happy puppy again.
That could have been the end of my blog post.
However I wrote this as a preamble to what I am going to show you next – how to setup your own Sync Server at home (or at work).
When I was contemplating the (dis)advantages of using Mozilla Sync, I was not too fond of the idea that the Sync Server which Firefox will use by default is hosted by Mozilla. Even though your data is supposedly being encrypted with your unique sync key before it gets uploaded to Mozilla, it feels awkward to see all your passwords and browsing data being transfered to a 3rd party. Furthermore, the geek in me was just dying to setup a sync server of my own.
Luckily, Mozilla is open as always and makes the Open Source Sync Server code available to everyone who wants to play with it. And setting it up proved to be not difficult at all. Note: Mozilla have re-designed and re-written the original Sync Server (it was called “Mozilla Weave Server” at the time, and was almost impossible to setup… I tried, back in october 2010). It’s now written in Python, and the steps to create your own Sync Server are described on this page: http://docs.services.mozilla.com/howtos/run-sync.html. That page is what you should follow. The rest of my article tries to bring more detail to the process of configuring your Slackware box to become a real Sync Server.
Of course you want to install this on a Slackware server! There are some explicit dependencies as mentioned on the Mozilla page I referred to earlier, but python 2.6 and sqlite3 are already suppied by Slackware, and I created a package for virtualenv (for Slackware 13.1 and 13.37) which you can download here: http://slackware.com/~alien/slackbuilds/virtualenv/ .
Note (20120627): it looks like installing this virtualenv package is no longer needed, but I did not check this. A comment from Old_Fogie at the bottom of this package indicates that virtualenv is downloaded automatically during the “make build” you will run as part of the next step in my procedure. My suggestion is to skip the installation of my virtualenv package at first, and see what happens you run the “make build”. If you get an error about missing virtualenv, then install that package and re-run “make build”…
The Sync Server code is hosted in a Mercurial repository and since Slackware ships a Mercurial client, that code is easy to check out to a local directory where you continue to build the server binary:
$ mkdir -p /usr/local/server-full $ cd /usr/local/server-full $ hg clone https://hg.mozilla.org/services/server-full/ . $ make build
Now, this server is ready to run, but unless you make some configuration changes it is not going to do you much good.
By default, the server stores your sync data in a sqlite database, which is fine with me (you can use a real MySQL server database if you want), but it will create the sqlite database in /tmp which is entirely inappropriate. Further down I will show you what to modify so that your clients’ sync data go somewhere less “volatile”.
The python-based Sync Server can run all by itself on a configurable port number (5000 by default) which is just fine if you are its only user in your little LAN. But even Mozilla advises to put a real webserver between you and the Sync Server. For this I used Slackware’s apache webserver. This means, you will need one additional package, mod_wsgi which you can download here: http://slackware.com/~alien/slackbuilds/mod_wsgi/ . I created a separate system user and group (weave:sync) which is going to run the Sync Server so that it can not interfere with Apache if anything goes crazy:
# groupadd sync # useradd -g sync weave
If you omitted the creation of the homedirectory for this “weave” user account, then create it like this:
# mkdir /home/weave # chown weave:sync /home/weave # chmod 711 /home/weave
You then create a new file “/etc/httpd/extra/httpd-wsgi.conf” with the following content (adapt to your actual sync server hostname):
<Directory /usr/local/server-full> Order deny,allow Allow from all </Directory> <VirtualHost *:80> ServerName sync.myserver.net DocumentRoot /usr/local/server-full WSGIProcessGroup sync WSGIDaemonProcess sync user=weave group=sync processes=2 threads=25 WSGIPassAuthorization On WSGIScriptAlias / /usr/local/server-full/sync.wsgi CustomLog /var/log/httpd/sync.myserver.net-access.log combined ErrorLog /var/log/httpd/sync.myserver.net-error.log </VirtualHost>
As you may notice, this tells that a script called “sync.wsgi” is going to be spawned as user “weave:sync” whenever a client connects to http://sync.myserver.net . You will have to modify your apache main configuration too (i.e. “/etc/httpd/httpd.conf“), in order to make it load the mod_wsgi module and use its configuration file. Add these lines to the httpd.conf:
# Activate the WSGI module (serving python binaries to the web): Include /etc/httpd/extra/mod_wsgi.conf # Act as a reverse proxy to the Mozilla Sync server: Include /etc/httpd/extra/httpd-wsgi.conf
… followed by an Apache restart (you can run “apachectl configtest” first, to check for mistakes in the modified configuration if you can’t affort your webserver to go down thanks to a typo)
Now comes the part where you configure the Sync Server configuration files. Take a look inside your directory “/usr/local/server-full” where the Sync Server code has been downloaded and built. You will find the aforementioned “sync.wsgi” script there. Open the file in an editor. Read through the file and change it to something which you find better suited. For instance, I changed “/tmp” to “/var/tmp” in the line “os.environ[‘PYTHON_EGG_CACHE’] = ‘/tmp/python-eggs'”:
os.environ['PYTHON_EGG_CACHE'] = '/var/tmp/python-eggs'
Next, edit the file “development.ini” (that name is configured in “sync.wsgi” in case you want to give it another). In that file, I changed the location of another log file so that it gets written in “/var/tmp”. The line “args = (‘/tmp/sync-error.log’,)” changes to:
args = ('/var/tmp/sync-error.log',)
This “development.ini” file also mentions the filename of the actual sync configuration (yeah, my idea too was that this level of nesting configuration files seems unwanted): it has the name “etc/sync.conf” relative to the server’s directory. It requires some editing! For instance, look for the two lines that look like this and change the path name to a safer location in your filesystem! I decided to create the SQL database file in the “weave” user’s own homedirectory, so I changed the line “sqlite:////tmp/test.db” to:
sqluri = sqlite:////home/weave/my_weave_server.db
The default value for the quota assigned to each sync account is too low (it’s set to 5 MB). Mozilla states that 99.9% of all clients which connect to its public server use less than 25 MB of quota, so use that value. The line “quota_size = 5120” changes to:
quota_size = 25600
Then, look for the following line and make absolutely certain that it is changed to the Apache http:// URL of your Sync Server, i.e. the URL that the sync clients are going to use:
fallback_node = http://localhost:5000/
You can play with other options in this file to see what they can offer. For instance, I disabled “captcha” and did not enable SSL, but you may decide that this is important to you.
Don’t forget to restart your apache server if you have not yet done so.
And now it is time to tell your Firefox browser to use this new service!
- Select “Tools > Setup sync…”:
- In case this is the first time you talk to your Sync Server, you need to setup an account, so click the button “Create a New Account“. This opens a new dialog box where you can type your email address (this is your account name), associated password and then in the dropdown do not select “Firefox Sync Server“… instead, you choose “Use a custom server“, it allows you to enter the URL for your brand-new private Sync Server. Once the account has been created, Firefox will show you the sync key it will be using to encrypt your data. That key is important, you will need it if you want to add more computers to your sync account. And that’s basically it – the sync process will start immediately:
- Else if you already have a Sync Account and want to add a second computer to it, you can click on “Connect” in the first dialog which will open a new dialog. There are two ways to add this computer to your existing sync account. Either enter the code which is being shown on another computer which is already syncing to the server; or else you click on “I don’t have the device with me“. This is what you would usually do I guess… it is combersome to search for another computer or hand-held if all you need to do is know your sync key:
- If you know your sync key you can enter that in the following dialog, along with your account details and the server URL:
- The sync key which is associated with your sync account can be retrieved at any moment should you have forgotten to write it down. Just go to a computer which has been setup for Sync, and open your Firefox browser’s Preferences at the “Sync” tab. Under “Manage your account” you will find a menu item to display your key (as well as a way to change your account’s password). More help can be found at http://support.mozilla.com/kb/what-firefox-sync .
Have fun! Eric
Very interesting post, I will probably try to install this service soon.
Thanks a lot Eric!
Nice! Thanks for the builds.
Thanks for libreoffice, Eric.
Nice job man!
Keep it UP!
Alas, I can’t seem to get mine working 🙁
I keep getting error messages about “‘sync.wsgi’ cannot be loaded as Python module.”
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] mod_wsgi (pid=4902): Target WSGI script ‘/var/mozilla-sync/server-full/sync.wsgi’ cannot be loaded as Python module.
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] mod_wsgi (pid=4902): Exception occurred processing WSGI script ‘/var/mozilla-sync/server-full/sync.wsgi’.
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] Traceback (most recent call last):
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/sync.wsgi”, line 74, in
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] application = loadapp(‘config:%s’% ini_file)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 203, in loadapp
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return loadobj(APP, uri, name=name, **kw)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 224, in loadobj
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return context.create()
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 617, in create
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return self.object_type.invoke(self)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/loadwsgi.py”, line 109, in invoke
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] return fix_call(context.object, context.global_conf, **context.local_conf)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/util/fixtypeerror.py”, line 57, in fix_call
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] val = callable(*args, **kw)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/deps/server-core/services/baseapp.py”, line 342, in make_app
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] app = klass(urls, controllers, params, auth_class)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/deps/server-storage/syncstorage/wsgiapp.py”, line 102, in __init__
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] auth_class)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] File “/var/mozilla-sync/server-full/deps/server-core/services/baseapp.py”, line 125, in __init__
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] signal.signal(signal.SIGTERM, self._sigterm)
[Mon Jan 02 02:29:00 2012] [error] [client 96.53.116.78] ValueError: signal only works in main thread
Looking at the path names (/var/mozilla-sync/server-full/lib/python2.6/site-packages/paste/deploy/) this seems to be a pre-packaged mozilla-sync server for some other distribution than Slackware. If that is the case, you should talk to the one who created the package.
What I used was the code available at https://hg.mozilla.org/services/server-full and the Python 2.6 which is part of Slackware. No issues there.
Eric
Hello Eric !
Thank you for this guide. I just ran thru this tonight and have the following comments:
There appears to be an error in this howto at:
Quote: You then create a new file “/etc/httpd/extra/httpd_wsgi.conf” with the followin….
Because later in the guide we are instructed to edit httpd.conf with:
Quote: Include /etc/httpd/extra/httpd-wsgi.conf
The error is in the “_” and it should be the same as what we put in the httpd.conf file, that is, it should be a “-” , eg: “httpd-wsgi.conf”
So the line in the howto article should read:
You then create a new file “/etc/httpd/extra/httpd-wsgi.conf” with the followin….
Regarding dependencies, the make process downloaded the following files, note the version of virtualenv is greater than the version of the package we are advised to install. Which is of no surprise as quite some time has passed since this article was written. So I’m not sure if we need to install the virtualenv package anymore ?? as the make process will download as needed.
Here is an abbreviated version of the output from make process:
Downloading distribute-0.6.27.tar.gz
Downloading MoPyTools-3.2.tar.gz
Downloading Paste-1.7.5.1.tar.gz
Downloading PasteScript-1.7.5.tar.gz
Downloading PasteDeploy-1.5.0.tar.gz
Downloading flake8-1.3.1.tar.gz
Downloading Distutils2-1.0a3.tar.gz
Downloading virtualenv-1.7.2.tar.gz
Downloading pypi2rpm-0.6.3.tar.gz
Downloading argparse-1.2.1.tar.gz
Downloading PasteDeploy-1.5.0.tar.gz
Downloading nose-1.1.2.tar.gz
Downloading WebTest-1.3.4.zip
Downloading WebOb-1.2.zip
Lastly, I suggest appending to the article for those, that are following your recommendations to the letter:
mkdir /home/weave
chown /home/weave weave:sync
touch /home/weave/test.db
chown weave:sync /home/weave/test.db
Perhaps we should chmod 0700 /home/weave as well ? I haven’t tested that yet.
Thanks again Eric ! Great job!
Hi Old Fogie,
Those are all valid and useful comments, thank you! I have made sure to implement them into the main article, and made some other enhancements as well.
Just hours ago, I installed the new native Firefox for Android on my smartphone and used my sync server to be up and running very quickly. The server is a big time saver.
Cheers, Eric
I ran into an issue where I get “Already in Use”. This after I deleted the DB file which I confirmed with sync devs that’s the only location the account name is stored.
I was wondering if you had this issue and what would be causing it?
hmm.. when I try to enter my brand new sync server url (as sync.domain.lv, with or without http://) into respective seamonkey registration dialog, I see “please enter valid server URL”.
Is there anything to do to get around?
Janis
sorry, as usually with me – solved by myself after asking a question (problem lied with permissions to log file)
The only thing remains unclear – if I add a device to the personal sync server’s account – how browser deciphers all the information necessary for connection using just 12 alpha-numerical characters? Or some (a lot of ) information goes to Mozilla anyway?
I have followed step by step in this tutorial, everything is describe, how to install and configure own Firefox Sync Server (Weave) with MySQL.
http://terminal28.com/how-to-install-and-configure-own-firefox-sync-server-weave-debian/
slackware’ish startup script would be very welcome
Hi Janis, a startup script would not be needed because the sync server is configured to be part of your Apache service.
Eric
Hi, Eric!
It seems it’s not my case – I did (I think so) as you write, but anyway I have to run sync server binary to get synced
Hi Janis
The point is that the definition line “WSGIScriptAlias / /usr/local/server-full/sync.wsgi” in the “/etc/httpd/extra/httpd-wsgi.conf” file triggers the start of the sync server when a client connects.
Eric
Hi!
I re-checked – i have the definition, but still have to run bin/paster by hand. Respective the http server logs are empty
Might be interestng to find out if this is a Apache 2.2 versus 2.4 difference. I run this on Slackware 13.37 using apache 2.2, while Slackware 14.1 has Apache 2.4.
Eric
yes, I have 2.4.3 installed on that server
Janis There is a slight change with Apache 2.4 that you need to make in the configurations you can see the changes in the slack docs wiki.
FYI that SlackDocs location is http://docs.slackware.com/howtos:network_services:weave#configuration
Eric
I have a config like seen in wiki from the very beginning. Despite that the service is not spawned on reqest for sync and there are NO error messages anywhere – virtual server logs are empty, sync-error.log is not even created.
As I wrote before – only bin/paster helps.
May be it doesn’t like the fact I use mysql as DB?
or some over-restrictive apache cfg:
ServerLimit 6
StartServers 2
MaxClients 20
MinSpareThreads 10
MaxSpareThreads 20
ThreadsPerChild 10
MaxRequestsPerChild 700
KeepAlive On
MaxKeepAliveRequests 200
KeepAliveTimeout 5
?
As Mozilla stopped to support old Sync (at least in latest Firefoxes), the topic should be re-visited and reviewed. Curently I ahve no other Idea than to set up the old-style sync server for personal use.
I still have nothing to write about an alternative for the old-style Weave Sync. Mozilla has documented the technical setup here: https://docs.services.mozilla.com/howtos/run-sync-1.5.html
But I am still unresolved whether I would want to host just the storage or go full-fledged and also host a Firefox Account server.
The old sync will not work if you install a new computer with a recent Firefox. So what I usually do: Install an older version of Firefox, configure the Sync using the above article, and then upgrade to the latest Firefox. Old-style sync will remain functional using your private Sync server.
According to https://support.mozilla.org/en-US/kb/firefox-sync-upgrade-frequently-asked-questions the old Sync system should have been de-activated on 30-sep-2015 but so far the browser is not complaining.