You will not be surprised if I tell you that I install Slackware a lot. Inbetween the public updates to slackware-current which you can read in the ChangeLog.txt there is usually a lot of private testing.
For that purpose I keep a local mirror of the slackware-current tree. I have written a script that keeps this local copy synchronized, and also creates the ISO images which I use for the installs. The script is widely used by other people as well, because there are no official “slackware-current” ISO images available for download. It is much more economical to create your own ISO images when you already have a local mirror of all the packages.
The script is appropriately called “mirror-slackware-current.sh“. If you want to know what it can do, just run it with the “-h” parameter:
$ /usr/bin/mirror-slackware-current.sh -h
-----------------------------------------------------------------
$Id: mirror-slackware-current.sh,v 1.70 2009/04/14 10:24:48 root Exp root $
-----------------------------------------------------------------
Usage:
/usr/bin/mirror-slackware-current.sh [OPTION] ...
or:
SLACKROOTDIR=/your/repository/dir /usr/local/sbin/mirror-slackware-current.sh [OPTION] ...
The SLACKROOTDIR is the directory that contains the directories
slackware-<RELEASE> and slackware-<RELEASE>-iso
Current value of SLACKROOTDIR : /mirrors
You can change the script defaults in a file 'mirror-slackware-current.conf'
The script's parameters are:
-h This help.
-b <number> Limit bandwidth usage to <number> KBytes/sec.
-c Check for newer version of this script.
-e Use 'boot-load-size=32' instead of the value 4.
(32 is a more standard value, but a value of 4
will let the ISO boot with old 'broken' BIOSes).
-f Force sync and the creation of new ISO image(s)
even if no update of the ChangeLog.txt was found.
This is how you resume after an aborted attempt.
Note: this will also create any missing local
directories needed for the mirror.
-i Only generate ISO images from our local copy;
do not attempt to contact the remote server.
-l <localdir> The root directory where you keep your local
Slackware mirror; this directory contains
slackware-<RELEASE> and slackware-<RELEASE>-iso
-m <uri> The rsync URI that you want to use instead of
the script default. Example:
-m mirrors.tuxq.com::slackware/slackware-current
(no trailing slash!)
-n Only show the changes in the ChangeLog.txt
but don't sync anything and don't generate ISOs.
-o <iso_type> The type of ISO that you want to generate.
iso_type can be one of:
CDROM: produce 3 CDROM images (KDE on CD3);
DVD : produce a single DVD image;
ALL : produce CDROM and DVD images;
NONE : produce no images at all (just sync).
The default iso_type is CDROM.
-p Remove old ISO's before building the new ones
(in case you're suffering from low free space).
-r <release> The release ('current' by default); use '-r 12.2'
if you want to mirror and image slackware 12.2
-q Non-verbose output (for cron jobs).
-s Additional ssh options, in case rsync needs to
login to the remote server using ssh. Example:
-s "-l alien -o IdentityFile=/home/alien/.ssh/id_rsa"
-v Verbose progress indications.
-w Write a .conf file containing script defaults.
It will be created in the script's directory,
as 'mirror-slackware-current.conf'
-X <xfile> File 'xfile' contains a list of exclude patterns
for directories that you do not want mirrored.
Note: this will override the default exclusion of
the 'pasture' directory so if you still want that
excluded, add it explicitly to the file 'xfile'.
If your intention is *not* to exclude '/pasture'
from the mirror, use '-X none'.
An example: I have defined a cronjob which checks a Slackware mirror server once a day, and if the script detects a change in the ChangeLog.txt it will bring my local mirror back in sync, and creates a bootable DVD ISO image for me. If no changes are found, the script does not produce any output, but in case of an update it will show it’s actions on standard output. The cron daemon will mail the output of any job to the user account which runs the script, so in the morning I will find a comforting message in my inbox:
Date: Sat, 28 Mar 2009 15:44:37 +0100
From: root@darkstar
To: root@darkstar
Subject: cron: /usr/bin/mirror-slackware-current.sh -q -X none -o DVD
0a1,17
> Fri Mar 27 20:30:48 CDT 2009
> a/module-init-tools-3.6-i486-2.tgz: Patched modprobe to not complain if
/sys
> is not mounted. This is none of modprobe's business. ;-)
> l/glib2-2.18.4-i486-1.tgz: Upgraded to glib-2.18.4 and patched.
> This library has been patched to fix overflows that are
> possible security holes.
> For more information, see:
> http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-4316
> (* Security fix *)
> l/libxml2-2.7.3-i486-1.tgz: Upgraded to libxml2-2.7.3.
> xap/mozilla-firefox-3.0.8-i686-1.tgz:
> Upgraded to firefox-3.0.8.
> This fixes some security issues.
> For more information, see:
> http://www.mozilla.org/security/known-vulnerabilities/firefox30.html
> (* Security fix *)
> +--------------------------+
Sat Mar 28 15:22:06 CET 2009 [17616]: ChangeLog.txt has been updated,
starting mirror of slackware-current.
*** Using rsync.osuosl.org::slackware/slackware-current ***
Sat Mar 28 15:24:09 CET 2009 [17616]: Done mirroring slackware-current
(exit code 0).
Sat Mar 28 15:28:04 CET 2009 [17616]: Creating DVD ISO image for
slackware-current...
Sat Mar 28 15:38:27 CET 2009 [17616]: DVD ISO created (exit code 0) ...
Sat Mar 28 15:38:37 CET 2009 [17616]: Computing MD5 checksums of the ISO's
(time consuming).
Sat Mar 28 15:44:31 CET 2009 [17616]: Resulting ISO files:
-rw-r--r-- 1 root root 4479451136 Mar 28 15:38
/mirrors/slackware-current-iso/slackware-current-install-dvd.iso
Sat Mar 28 15:44:31 CET 2009 [17616]: Done!
This is the command line which I added to my crontab:
# Keep the Slackware -current tree in sync:
22 5 * * * /usr/bin/mirror-slackware-current.sh -q -X none -o DVD
The script used to create 2 CDROM ISO images if you wanted those (leaving out the kdei/ series, as well as the packages in extra/), but lately, the size of Slackware has increased so much that the packages no longer fit on two CDROMs. I have modified the script so that it will now produce 3 CDROM ISO images (with the kde/ series on the third CDROM) if you want to burn CD’s.
TIP: If you want to change the default values of the script, do not edit the script itself. Instead run “mirror-slackware-current.sh -w” which will create the file “mirror-slackware-current.conf” in the same directory as “mirror-slackware-current.sh”. If you edit the variable values in “mirror-slackware-current.conf” the script will use those.
Have fun with this script,
Eric
Tip:
Include the possibility of filtering by language.
rsync -vazP \
–delete-after –partial-dir=.partial-dir \
–exclude=pasture –exclude=source \
–delete-excluded \
–filter=’+ */’ \
–filter=’+ **/aspell-0*’ \
–filter=’+ **/aspell-pt*’ \
–filter=’- **/aspell*’ \
–filter=’+ **/kde-i18n-pt_BR*’ \
–filter=’- **/kde-i18n*’ \
–filter=’+ **/kde-l10n-pt_BR*’ \
–filter=’- **/kde-l10n*’ \
–filter=’+ **/koffice-l10n-pt_BR*’ \
–filter=’- **/koffice-l10n*’ \
Tks!
great script
Hi, there
Is it possible to get a script for generating the meta files and packages.txt a.s.o. for a local repository?
is there any howto arround to create a repository for txz files?
I have exactly such a script. I use it to create the meta data for my own repository at http://slackware.com/~alien/slackbuilds/ including a ChangeLog.txt and RSS feed. Get the script at http://slackware.com/~alien/tools/gen_repos_files.sh and run it with the “-h” parameter to get help on its usage.
About creating “.txz” packages, that is not hard at all. If you tell makepkg to create a package ending in “.txz” then it will automatically apply the correct (xz) compression. If you use SlackBuild scripts to create your packages, you only need to edit the line for “makepkg” and ensure that the package name ends on “.txz”.
It is entirely possible to have a repository with a mix of “.txz” and “.tgz” packages. The script I refered to is able to create proper PACKAGES.TXT and other meta files for such a mixed set.
Cheers, Eric
i wrote a similar script years ago called slackmon, think it’s still on sourceforge actually… your one is better tho =P
Hi Eric. great script as always. I’m mirroring slackware with your script and i want to build the disk with my own packages i have built such as transmission, xclip, skype, evince. etc. Have tried to edit the script but unsuccessfully. Is there any chance you would bother with this, or help me out? 🙂
thanks for all the hard work and support! Cheers
@pupit –
Sure, you can contact me on my alien @slackware.com email or ask the question here on the blog page.
Eric
Hi Eric/Alien
First of: thank you for all your contributions to the slackware scene, google sent me numerous times to your wiki when i was looking for an answer. Especially your wiki articles on vde and qemu helped me out a lot!
I was truggling with this cool script because i was this getting this cryptic error from rsync:
Unexpected remote arg: rsync://taper.alienbase.nl/mirrors/slackware/slackware-current/
rsync error: syntax or usage error (code 1) at main.c(1201) [sender=3.0.6]
(As you can see i’m using your mirror after i read this: http://www.linuxquestions.org/questions/slackware-14/slackware-rsync-mirrors-750959/ 😉 )
Turns out i was trying to exclude “source”, “slackware/kde” and “slackware/kdei” the wrong way by just adding them to the exludes in the conf file like:
EXCLUDES=”–exclude pasture source slackware/kde …..”
From the rsync manpage/FILTER RULES section:
Note also that the –filter, –include, and –exclude options take one rule/pattern each. To add multiple ones, you can repeat the options on the command-line, use the merge-file syntax of the –filter option, or the –include-from/–exclude-from options.
Maybe you could mention that in the conf file?
Cheers, another happy slacker from Eindhoven.
PS: what is a good mirror for me to use?
Sorry, i’ve just found out the script mentions some good mirrors….nevermind the question.
Very ignorant about this rsync business.
in /usr/local I tried
./mirror-slackware-current.sh -f
and it is grinding away, but where do the results go? What do I do with them? My goal is one DVD or multiple cds with the latest and presumably greatest Slackware.
John Culleton
May have answered my own question. Discovered stuff in
/home/ftp/pub
added -o DVD to the command. kept -f as well.
Onward and upward
Just came to know that you are having this facilities as well. i was trying to figure out how to install -current as there was none in most of the mirror i am familiar with. This makes it much easier for installation and its up-to-date version save lots of time, as there is less upgrading/updating needed so much. Thanks!
Hi Eric,
Thx for the script, using it today 🙂
Thx also to DaaNMaGeDDoN for posting about how to exclude multiple directories in rsync, I was getting the same error until I followed his suggestion like:
EXCLUDES=”–exclude pasture”
EXCLUDES=”–exclude source”
I have a question: would it be possible to add an option that shows the progress of rsync? For example in my scripts that sync using rsync I use the following options:
-avP
-a, –archive archive mode; equals -rlptgoD
-v, –verbose increase verbosity
-P same as –partial –progress
Regards,
Aaditya
The method mentioned in my previous comment about excluding certain directories is incorrect, ended up downloading lots of unwanted stuff 🙁
It can be like:
EXCLUDES=”–exclude pasture \
–exclude source”
Aaditya, there is a “-v” option to the script, does that not do what you want?
The “DVD_EXCLUDES” variable example a few lines further in the script might have given you a clue about the fact that “EXCLUDES” can contain multiple space-separated exclude parameters…
I dont think that the -v option shows progress, will give it a try next time.
Yeah, did not look closely I guess.
Thanks for the script. I’m using it to add these nearly 2 years of 14.2 patches into the install DVD for offline re-install of my router/firewall PC. (wish we would have had official 14.2.1/14.2.2 DVD though with them already integrated) Suggestion for the script: run “readlink -f” or something like that on the value to the -l parameter. Things like “./mirror-slackware-current.sh -l . -r 14.2 -n” or “./mirror-slackware-current.sh -l ../mirror -r 14.2 -n” now have the directory ${SLACKROOTDIR} points to change after the “cd” the script does when using such relative paths after the “-l”, causing it to error… (same seems to apply to the -X option)
Marco I can fix that in the script.
Hi, AlienBob!
Please add the option to the ARM port in the local mirror script, we need to follow the staff of Pi.
Your script is very complete, just missing this to make me happy.
Thanks in advance!
Hugs
Antonio.
On 30 March you asked exactly the same in the Feedback page, that same day I answered “Antonio, I added ‘arm’ as a new supported architecture to the “-a” parameter. Please check if this does what you need.”
Are you unable to read my replies? Is my english so bad that you can not comprehend?
AlienBob!
I could not see his answer.
I tested the script and noticed the following error:
sent 141 bytes received 186,747 bytes 21,986.82 bytes / sec
the total size is 2,820,166,649 acceleration is 15,090.14
Sun Apr 22 15:41:36 -03 2018 [8173]: Slackwarearm current mirroring (exit code 0).
Creating a DVD ISO image for slackwarearm-current …
! mkisofs: File or directory not found. Unable to open isolinux / iso.sort classification files list
Sun Apr 22 15:41:36 -03 2018 [8173]: ISO DVD created (exit code 2) …
mv: could not get status of ‘/mnt/sda3/Slackware/slackwarearm-current-iso/.building-slackware-install-dvd.iso’: File or not found
Well there you have it. Ther ARM release does things a little different. According to http://arm.slackware.com/getslack/ it does not make sense to create a bootable ISO for slackwarearm, since there’s no device that can actually boot from a DVD. So a couple of files are not present in the ARM tree – everything that would be necessary to create a bootable ISO image.
So the script needs some more adjustments to make it work for ARM.
I will add that to my TODO but since it is a bit more work it will not be high on my priority list.
No problem!
It has already been a great job to include ARM in the script.
Health and peace!
Hugs