Geek Stuff

[Kruger]

Hey, what process do you use for backups?

I screwed up, royally...

I use rsync -raz --partial --delete --log-file

I liked delete since it maintains the backup and deletes files that I delete from the source.  

I had a hard drive start to fail about six months ago, it was my music library.  After I did some patching and updated a new kernel then rebooted, I just didn't check for a few days and that drive didn't mount.  I thought oh shit, but was able to mount it manually.  I was thinking it just didn't spin up in time and just wasn't available.  But restarting the server a few times and it was all good, no issues with that drive.  I checked the backup logs and because it wasn't mounted the backup skipped over all of that media and left it alone.  Ok cool, I thought.  If/when the drive fails I'll be ok...

until....

It failed again on the 16th.  I usually only notice it when I try to stream some music and that's usually just when I'm driving.  So I check in on everything and that drive had a catastrophic failure, it's dead... dead... dead...  Never coming back.  Ok, no issues, I'll just order a new drive and restore, until I check the backup and all of that media is gone.  Looking at the logs it saw the empty html folder (that, the subfolders for subsonic and plex, and the music were all on that failed drive) and deleted the backup.  Oh crap...  

So now, I'm not sure how rsync works and determines it should delete or not.  The first time it left everything alone, maybe because the drive was there but it just wasn't mounted?  But when it failed completely the mount point was truly empty and so guh-bye...

Anyway, I'm just curious.  What process are you using to backup your server?

[Blaze]

Oh man, that sucks. I really know how you feel because things like that have happened to me more than once.

So to answer your question, I have a plugin on my OpenMediaVault NAS that is set to rsync my nas drives whenever I plug in my external USB hard drive. One of the settings I made sure was off was the delete option. I just couldn't trust it completely for reasons you experienced. Or what if the USB drive is acting up and the backup doesn't see the file so it deletes it off the server. I didn't want to take that chance. Instead, I'll just manually go through the USB drive afterwards and prune it as I see fit. But lately, since hard drives aren't that expensive - when compared with my time of pruning a hard drive versus just slapping in a new bigger drive is quicker.

Now, when I backup my main rig to the NAS, I do run rsync with the delete option - however I always do a dry run first and I have everything logged where I'll go through that to see what it's going to do. In my backup shell I run, this is a snippet of the commands I'm using:

backup_path_home='nas-vm::Blaze/home/'
current_date=`date +%Y%m%d`

rsync --dry-run \
--archive --update --verbose --human-readable --itemize-changes --progress \
--recursive --perms --delete --delete-excluded \
--exclude-from='/home/blaze/backup_exclude.txt' \
--log-file='/home/blaze/Backup/rsync-home-x.log' \
/home/blaze/ ${backup_path_home} 2>&1 | tee /home/blaze/Backup/rsync-dry-output-home-${current_date}.txt

With my first NAS I lucked out when a drive failed since it was raided. It was a nice feature to just slap in a new drive and have it get built. Shortly after I did build a new one and put the old drives in the archive since if one failed of "old-age" it was only a matter of time before the others did as well. I also do a S.M.A.R.T. test every night on the drives and an extensive test on them once a month. These are all automated and scheduled from the server and I get e-mailed a report. And when the backups are happening, I get a little on edge and will constantly check the rsync logs on the server to see what the progress is and then do some spot checks to make sure files are truly getting backed up and nothing is going awry.

The big takeaway from this would probably be to do a dry-run before actually backing up to make sure everything is going as planned.

[Kruger]

Ugh, I'm still so 90's....  I "planned" to build a nice new(ish) server with RAID and redundancy.  I just never got around to it, and it really bit me in the ass this time.  It looks like it was just the web server and media servers along with the music libraries that went away.  While it's just a pain, I still have all of those CDs so it'll just be time consumed to re-rip everything again.  The drive with all of my video on it was mounted into the html/video folder and since that didn't exist anymore, rsync just left the backup alone.  I think that's kind of what happened that first time when it didn't see the html folder it was left alone on the backup.

Anyway, I just did my testing when getting that set up and I could recover, I just never tested the drive failure part of it and so it was a wait and see sort of thing.  I just set it to run every night and check it daily for about a month then less frequently as time went on.  Until I try to listen to music...

I still like the delete feature, it keeps things "clean" while I rearrange tings on occasion.  But I might add some error checking to make sure there is content in the mount point before trying to backup, and maybe doing the backup without delete daily, and then once a month manually run a backup script with delete once I confirm it's all good.

Oh the joys of having a home network.  Now if only I can squeeze another hour or two out of my day for management...

[Blaze]

Now if only I can squeeze another hour or two out of my day for management...

Man if that doesn't just resonate with me to the fullest. The NAS is just one part of my home network. I have the Plex server, my DNS server and the router. The router is always fun since I'm running multiple networks and each one goes through a different VPN. Depending on the device, you could be going through New York, New Jersey or Montreal. Keeping on top of all those configurations can be a bear. And I rooted my wireless router and have OpenWRT installed on that. Keeping that maintained can be some work too - although, I've relaxed the hardening of its access a while back because fuck Apple and not exposing or offering a way to determine the MAC address on a device. I would have to set up a hotspot on my rig and log the connection to get the MAC address. Also because I've been playing a lot with wireless Raspberry Pi devices (and their clones) as well as some ESP32 controllers and not having to create an entry for each one in the router is a time saver.

I've also started fooling around with home automation - although, none of the software I've been test driving plays nicely within a VM. And I also am not a fan of Docker. What an absolute pain in the ass it is trying to manage those. Half the time I can't even get into the device because I don't know what internal IP the damn thing is using.

But if you're going to build a NAS, really go with a RAID system and if you can handle it, use RAID 5. With the new hardware, I ditched the separate hardware RAID controller and opted to go with software RAID. I don't recall if the motherboard is handling that or the NAS OS. Either way, I think the newer system is probably faster than my old RAID controller - plus I think the controller didn't do RAID 5, but instead RAID 3.

[Kruger]

Well, time being what it is, I'm just going to slap a new drive into the server and start ripping CDs during my work day (slide over to the media box, put a new CD in, find the correct track listing and let 'er rip)

Eventually, eventually there will be a NAS with RAID 5...