VDO Troubles

Once in a while, on very large VDO volumes, storage just dissapears.  In my case I had one 42TB volume that just refused to boot.  Here are some of the symptoms:

• Boot dropped single user into a shell
• "vdo start" says the vdo volume was already started, but there is no entry in /dev/mapper/.

In this case I retried "vdo stop" then "vdo start" a few times until the vdo volume became available.

Further observation implies that systemd kills the vdo startup process.  What happened was that vdo was interrupted on its long boot up---a timing issue.

A working solution for me was to include a timeout value in /etc/fstab for the particular vdo volumes:

# /etc/fstab
 /dev/vgsample/lvsample /somedirectory ext4 defaults,discard,x-systemd.requires=vdo.service,x-systemd.device-timeout=10min 0 0

Working so far, survived a few reboots.

JondZ 20181031

Clustering legacy websites with keepalived and haproxy.

We had a legacy website we could not get rid of, it was very stable on debian lenny 5.0.  After much grief I managed to compile and RPM version of php 5.2 which it needed --- not php 5.3, not 5. anything, it HAS to be 5.2.  Ok so, finally I was able to get it running in CentOS 6.

I also clustered the filesystem using gfs2.  I did not remember until  yesterday that I configured this server to also server SMB share.  It was like 5 years ago and users are still on it!  So I also had to use clustered samba (CTDB).  Finally I got it working again.

I was able to get this thing clustered using keepalived cookie tricks--basically haproxy tricks users browser into connecting to only one and the same backend webserver, but different users connect to different backends.  Thats basically Active/Active already.  I just finished and tested this right now even on a holiday weekend.

So what we have now is

---ext ip ----- frontserver1/frontserver2 ------- webserver1/webserver2

The webservers themselvfes also host mysql.  Basically a four-pack HA setup.  I tested rebooting nodes and the websites continue to work.

Its good unfortunately a lot of IP addresses were allocated.  about 7 instead of just one .  But everythings virtual so I guess thats ok.

JondZ

a day at the office, spare rh ha license

Ok so I finally got HA/GFS2 nodes at work.  I'm getting a lot of mileage out of the licenses I was allowed to use, more than I expected.  First of all, a RH "unit" actually counts as one half when ran as virtual machines.  This enabled me to run a 6-pack stack consisting of 2 routers (keepalive/haproxy), 2 fileservers (GFS2), and 2 PGSQL (Active/Passive nodes).

Today I discovered that the RH Resilient Storage (GFS2) license already includes HA (pacemaker) license.  I called RH support to verify and make sure.  So, I can take that license and build corosync Q devices.  Our 2-node clusters are going to be upgraded with Q devices. 

It is a nice day.

vdo in an lvm sandwich

I have finished migrating one of the backup servers that we have (where I work) from a standard LVM setup to one that has a VDO engine.  It turns out the compression ratio was 41% or something like 10TB physical to 25T logical blocks--or something like that if I undertood the vdostats output correctly.

Someday I might post a quick how-to here.  I already posted a maintenance document in out internal website so i am a little tired right now.

Anyhow what I ended up doing was an LVM-over-VDO-over-LVM setup like this:

--------------------------------------------------
LVM disks -- actual exposed disks -- /data1 /data2, etc
-------------------------------------------------------
VGS (actual usable volume group, vg1 for example)
-------------------------------------------------------
vdo1 (dedup/compression/zero-elimination)
-------------------------------------------------------
Logical Volume /dev/vg0base/disk1
-------------------------------------------------------
VGS (vg0base for example)
-------------------------------------------------------
PHYSICAL DISKS
-------------------------------------------------------

What I like about this (except for the added complexity) is that the scheme doesnt really change existing setups too much. Provisioning is done in a normal way starting from the vg1 volume group above. After set up I can pretty much forget about the lower levels and do the usual disk creation such as "lvcreate -n disk1 -L 1G vg1".

The advantages of this setup are as follows:

1. Physical disks can be added if needed. Unfortunately I had to write a maintenance manual (where I work) becuase all the elements of the stack need to be expanded as well.
2. The VDO layer is easy to expand. I wanted to over provision I can simply expand the logical size of vdo1. The upper layers will follow the size (though not automatically--maintenance manual again).

Since there was existing data already (of about 25T) I had to bootstrap from a small existing space and slowly expand the vdo layer.  Fortunately the backup data were contained in separate partitions (/volumes/data{1,2,3,4}) so I was able to incrementaly migrat them, though the whole process took me a week to accomplish.


jondz

host to host nc/tar tricks

Here are some tricks to transfer files between hosts using tar.  In the old days by habit I would just do something like (tar cvfp - ) | ( ssh host "cd /path/to" && tar xvfp - ) but here are even more simpler things, assuming you have open terminals on both hosts.

CASE 1: SENDER IS LISTENING
Here are some commands that will work; the sender command needs to be typed first then it will block until a consumer appears on the other side of the pipe.

sender% tar cvf - . | nc -l -p 1234
sender% tar cvf >(nc -l -p 1234) . # process substitution

receiver% tar xvf - < /dev/tcp/ip-of-sender/1234
receiver% tar xvf <( dd obs=10240 < /dev/tcp/ip-of-sender/1234)

The magic number 10240 is 512*20, or the default 20 512-block records of the tar command.

CASE 2. RECEIVER IS LISTENING

This is the case when you are on the receiving end and want to initiate a receive before switching over to the sending terminal.  Type the receiver command first and IO will block until there is a sender:

receiver% nc -l -p 1234 | tar xvf -

sender% tar cvf - . > /dev/tcp/host-or-ip-of-receiver/1234
sender% < /dev/tcp/host-or-ip/1234 tar cvf - .
sender% tar cvf >(cat > /dev/tcp/host-or-ip/1234) .

NOTES:
Sometimes it is necessary to add "-N" option to nc, depending on platform/distro.

Edward 20180228