August 7, 2010 Hatem Ben Yacoub

Shared OCFS2 partition on Ubuntu Server 10.04 x64

One of the applications that I’m working on uses archived documents, there is no NoSQL here, just plain tiff files with indexes in Oracle database. Everything related to document access, permissions, conversions, watermarking, security, encryption … is managed by the application itself. So I had to keep my cluster permanently connected to a SAN storage via HBA fiber cards.

In the beginning I opted for NFS as shared file system, then decided to go for OCFS2 which is open source, maintained by Oracle, and available under GPLv2.

Some Pros/Cons of NFS, GFS2 and OCFS2 (from Dublin OSS barcamp) :

NFS

  • Pro: standard, cross-platform, easy to implement
  • Con: Poor performance, single point of failure (single locking manager, even in HA)

GFS2

  • Pro: Very responsive on large data files, works on physical and virtual, quota and SE-Linux support, faster than EXT3 when I/O operations are on the same node
  • Con: Only supported with Red Hat, Performance issues on accessing small files on several subdirectory on different nodes

OCFS2

  • Pro: Very fast with large and small data files on different node with two types of performance models (mail, data file). Works on a physical and virtual.
  • Con: Supported only through contract with Oracle or SLES, no quota support, no on-line resize

First we need to install OCFS2 tools :

sudo apt-get install ocfs2-tools

There is another package ocfs2console that you want to install to configure the cluster via GUI, but since I’m using ubuntu server I’m skipping this to configure my cluster manually.

CFS2Console-300x192

 

Create on every node attached to storage /etc/ocfs2/cluster.conf

sudo vi /etc/ocfs2/cluster.conf

With the content below, only replace node1 and node2 with their respective names and IP for each node :

node:
name = node1
cluster = ocfs2
number = 0
ip_address = 10.10.0.0
ip_port = 7777
node:
name = node2
cluster = ocfs2
number = 1
ip_address = 10.10.0.1
ip_port = 7777
cluster:
name = ocfs2
node_count = 2

Now you reconfigure ocfs2-tools with the default values :

sudo dpkg-reconfigure ocfs2-tools

then restart services :

sudo /etc/init.d/o2cb restart
sudo /etc/init.d/ocfs2 restart

If your fiber card connected to your host/storage, and virtual disks created and presented you should run fdisk to see it :

$ sudo fdisk -l
Disk /dev/sda: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x02020202

the result have been truncated to show only one virtual disk, and you might see multiple /dev/sda, /dev/sdb, /dev/sdc… according to your configuration in addition to your local hard disks. What I have done is creating a 1TB partition that I will share between my two nodes :

$ sudo fdisk /dev/sda

In fdisk menu choose “n” for new partition, and choose your partition size according to your requirements. Then use “w” to write changes and exit.

Finally we create a ocfs2 partition table :

$ mkfs.ocfs2 /dev/sda

mount your partition :

$ sudo mkdir /archives
$sudo mount -t ocfs2 /dev/sda /archives

or you can add it to fstab to mount automatically on boot :

$ /dev/sda /archives ocfs2 _netdev 0 0

The _netdev option is used here to prevent the system from attempting to mount these file systems until the network has been enabled on the system.

You want to test your new partition, and you will notice that every file/folder created on node1 is automatically available on node2, and vise-versa.

Enjoy !

Tagged: , , , , , , , ,

About the Author

Hatem Ben Yacoub Energy Engineer, Entrepreneur, ICT & eGov Consultant with over 15 years experience. Independent Open Data expert.

(HBY) Consultancy