Convert LVM logical volume Raid 10 to Raid 5

Two weeks ago I was running out of space on my main raid matrix on my server. Server was configured and assembled few years back, and at that time I had about 700GB of data there so having RAID 10 of 4 1TB disks (that account for about 1,7TB of actual space for data) was enough. Especially when you count in a price of 2TB (or 4TB) NVME M2 disk back then.

But you hardly remove any data so in time it takes more and more of your hard disk space. And in this case adding another disk was not the quickest solution to a problem; quickest would be changing the raid type to get one 1TB of space more.

It is possible (in theory, according to LVM documentation). I just never done it and server is constantly in use so I did not want to go offline or login in single user mode to take volume or group offline first. Anyway here is how I did it.

  • first change RAID 10 to stripped.
  • convert stripped to raid5_n
  • convert raid5_n to raid5_ls
  • add 4th disk to raid

Convert RAID 10 to stripped

This is simple and immediate step. Just run:

lvconvert --type striped {{vg}}/{{lv}}

Of course replace {{vg}} and {{lv}} with actual name of your Volume Group and Logical Volume.

Convert stripped volume to RAID5_n

This is not immediate and LVM need time to convert volume. Of course time of this operation is very dependent on size of your disks and performance of your machine.

You need to run following command:

lvconvert --type raid5 {{vg}}/{{lv}}

Since this need time to be completed in the background you need to check from time to time progress of conversion. This can be done by running command:

 lvs -a -o name,copy_percent,devices {{vg}}

Look for value represented by Cp%Sync. When this is done it will 100.00.

Convert RAID5n to RAID5ls

Raid5_n in LVM is just a intermediate type of raid that is meant to be used to followup it to another type – it is not meant to be used for production scenarios. Just for conversion it is fine but if you want RAID5_ls is more appropriate. If you interested why, you can read about that in docs.

lvconvert --type raid5 {{vg}}/{{lv}}

This will by default convert logical volume to raid5_ls, which is more production ready. Again this will take time (and add 3 disk) so relax and check status from time to time running command:

 lvs -a -o name,copy_percent,devices {{vg}}

Add 4th disk to RAID5

By default converting stripped logical volume to RAID5 will create matrix of 3 disks. To add another one and have more space for you data you need instruct LVM to add another disk to matrix. It is possible running following command:

lvconvert --stripes 3 {{pv}}

This time LVM won’t choose physical device for you. You need to point out to the correct one, so change {{pv}} to name of your actual device. This should be something like /dev/sda (if you are using HDD or SDD disks) or /dev/nvmeXnY if you are using NVME disks.

This will take some time (again! but this is the last one!), but you can use your new space right away and LVM will synchronize all disks in the background so you do not need keep tabs on it.

Word of caution

One thing you need to be worry about: sometimes LVM is complaining about lacks of space between conversions, specifically converting stripped to raid5 may be problematic since it requires few extents for metadata (about 4mb on each disk if I remember correctly). If you have some another (not used physical device) it may use it for conversion. There is also possibility to use memory for conversion – but this is not persistent – and I am not sure what will be an outcome of such conversion to RAID10.

Generally for RAID conversions in LVM it is usually needed to have 4-6 extents of free space on each disk so make sure you leave some free space on each disk when you configure your logical volumes for first time – in example leave out 10 extents.

No Display Manager in Debian Trixie

Lately I decided to try Podman Quadlets. It seemed to be really interesting way to achieve similar results as with Docker and Compose, but it have nicer integration with the system (i.e. ability to see container in Cockpit along with other systemd services, or triggering container by socket, so less frequently used containers won’t be running all the time).

Everything went fine except for unavailability of buildah package in Debian stable apt channel (or the version that were needed for podman at least). Quick search on the internet and I found the answer is to reference this specific package from the testing channel of apt.

At this time I knew that I am going into problems willingly.

I was able to install buildah package and install podman and in few minutes have running container. Put my laptop to hibernation and went to sleep.

Next day however I was unable to boot my laptop correctly. Or it was booting but only to text mode, without Gnome Display Manager.

Starting display manager manually systemctl start gdm3, or gdmflexiserver or gdm3 were not doing any good. Also systemctl status display-manager was not helpful. All Gnome related services were dead, closed soon after the system started without any meaningful errors. journalctl also did not show any critical errors like missing GPU drivers or similar issues, but startx was working just fine by showing terminal emulator by using desktop application – so driver were fine.

In attempt to fix that by updating entirely to testing Debian, or Trixie i followed this article. But it updated several packages more and beside that did not help.

I thought that maybe changing to alternative Display Manager should fix that. Another web search showed that doing dpkg-reconfigure gdm3 should show configuration for choosing/changing display manager. But instead resulted in similar error (I am typing from memory).

gdm3 package was not installed correctly or some dependencies are missing.

At this point fix seemed straightforward: reinstall Gnome.

apt reinstall gdm3

proved to be successful by installing few Gnome packages like i.e. gnone-shell, after that desktop automatically were started and Debian switched from text based terminal to Gnome login screen.

PS: strangely I was able to install podman on another machine running also Debian bookwork without a problem.

Monitoring vs Observability

What is the difference between monitoring and observability?

Monitoring is process of collecting live events from your application, traces, debug messages, warnings and exceptions. The same process may involve sending those events to some storage or application that analyze or visualize those messages. I.e. in asp.net core applications monitoring of application involves calling a ILogger and sending those logs or critical exceptions to AppInsights. Monitoring may also involve activity of development team members to active watch those logged messages for any potential errors or problems that may impact availability of production environment.

Observability is on the other hand is analyzing messages produced by monitoring process to understand behavior of application and reason about it internal state, if this state is expected (correct behavior, ‘application works ok’) or may be unexpected (erroneous state, database is down, endpoints returns 500 etc). Typically involves dashboard with several metrics from monitoring, exceptions, response times, number of not OK responses in time, long database queries, number of active users etc (i.e. App Insights dashboard in asp.net core). Monitoring process of an application (as human activity) is typically using such observability dashboard.

So what is the difference between Observability and Monitoring? Observability is possible because Monitorin of an application was implemented. If metrics and logs would not be collected and stored system is not observable.

Basically Monitoring is collecting logs and metrics, while Observability is feature of the system enabled by such process.

Solid

  • The single-responsibility principle – piece of software should have only one responsibility
  • The open–closed principle – things should be open for extension but closed for modification
  • The Liskov substitution principle – changing interface implementation should not brake software
  • The interface segregation principle – it is better to have more specialized interfaces than single big one
  • The dependency inversion principle – dependencies should be abstracted via interfaces

Further read:

https://en.wikipedia.org/wiki/SOLID

.NET/C# interview questions