I had a filesystem fill up and kill a job that I rather didn't want killed last night. I couldn't find a program to automatically grow lvm-hosted filesystems if I don't notice they're filling up and do something about it.
So I wrote a terrifyingly abhorrent bash script this morning to do it. Next meeting I'm at, remind me and I'll show it off in all its terror, and maybe someone can convince me there's a better way, or point directly to the program I should have found before writing my own.
And maybe, just extending this offer, will push me to finish cleaning up the edge cases in my script and properly finish it to avoid the embarrassment of being human in front of other humans.
LVM.conf has a number of autoextend options documented - at least on my Fedora box. Weirdly I don't see any references to this in the lvm man page, but only in the lvm.conf file itself.
Did you tie your script to inotify or udev rules or auditd events or anything like that?
On Thu, 2024-03-07 at 09:13 -0600, Billy Croan wrote:
I had a filesystem fill up and kill a job that I rather didn't want killed last night. I couldn't find a program to automatically grow lvm-hosted filesystems if I don't notice they're filling up and do something about it.
So I wrote a terrifyingly abhorrent bash script this morning to do it. Next meeting I'm at, remind me and I'll show it off in all its terror, and maybe someone can convince me there's a better way, or point directly to the program I should have found before writing my own.
And maybe, just extending this offer, will push me to finish cleaning up the edge cases in my script and properly finish it to avoid the embarrassment of being human in front of other humans. _______________________________________________ KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug
I count two, perhaps 3 if you include VDOs, but I don't think any of them really apply to me.
lvm.conf sure is verbose with its commentary! Curiously, there's a manpage for lvm.conf in Debian, but it seems only to describe syntax, not the full array of options.
I have used incrond/inotify to react to fs activity before. But I didn't think they had "awareness" of how full a fs is. And I don't want my script running every time any file is edited.
While probably not perfect, I used an while true loop with a dynamic sleep duration at the end that varies depending on how full the fullest filesystem is. Systemd makes sure my loop is always running, and my loop makes sure it's usually sleeping. When it isn't sleeping, it's checking, alerting, or acting.
I had not considered udev or auditd to react to FS usage. If they can kick off a script when an FS reaches x percent free or x gb remaining that would be an ideal choice.
In my script, a .conf file defines warning and action thresholds, and an action command. I have a separate script I use for the command that "Adds GBs to LVs" simply a wrapper to lvextend.
I've been harvesting a large amount of data lately, and these scripts have already saved me from waking up to an interrupted collection process several times.
The references I found to autoextend in lvm.conf were to automatically extend *snapshots* and *thin pools* and *vdos. not regular lvs*. I suppose with thin pools I could tell the filesystem that it's got 5000000000 TB to work with on a thin LV that only uses extents as they're written to, and then lvm would expand the thin pool as needed. That might accomplish the same thing. ---But something feels wrong to me about lying to the FS driver about how many blocks it has to work with.
I suppose 'thin pools' are sort of what I've recreated. But at least I know with what I made, I could reduce the FS size at a later date, and shrink the lv back down. I'm not sure I can say the same for a thin pool. I suspect I'd need to double my disks and make a fresh LV to move data to or buy another server.
It would sure be nice if lvm and ext4 played well enough together that ext4 could request more extents when it got full. When you wrote about lvm's autoextend options, this is what I was hoping for. But I don't think it's the case at all.
On Fri, Mar 8, 2024 at 8:48 AM xeniphon@gmail.com wrote:
LVM.conf has a number of autoextend options documented - at least on my Fedora box. Weirdly I don't see any references to this in the lvm man page, but only in the lvm.conf file itself.
Did you tie your script to inotify or udev rules or auditd events or anything like that?
On Thu, 2024-03-07 at 09:13 -0600, Billy Croan wrote:
I had a filesystem fill up and kill a job that I rather didn't want killed last night. I couldn't find a program to automatically grow lvm-hosted filesystems if I don't notice they're filling up and do something about it.
So I wrote a terrifyingly abhorrent bash script this morning to do it. Next meeting I'm at, remind me and I'll show it off in all its terror, and maybe someone can convince me there's a better way, or point directly to the program I should have found before writing my own.
And maybe, just extending this offer, will push me to finish cleaning up the edge cases in my script and properly finish it to avoid the embarrassment of being human in front of other humans. _______________________________________________ KCLUG mailing list KCLUG@kclug.org http://kclug.org/mailman/listinfo/kclug