UNMOUNT THE RAID
REMOVE ENTRIES FROM FSTAB
REMOVE NFS/SMB SERVERS
DO NOT ACCESS RAID WITH WRITE PERMISSIONS
GET SPARE STORAGE IF YOU DONT HAVE IT
This was quite the Experience!
The setup :
4x 5TB HDD presented to a Ubuntu Server 14.04 VM inside an esxi host via MAPPED_RAW_LUN in a MDADM RAID5.
I was trying to delete a user from ANOTHER VM, and I used a command deluser --remove-all-files. I had no idea it would go into my NFS mount and delete files that had been created by anoter VM with a the same username. This probably only happened because all my Production VM's are from a "Basic Template" so same UID/GID etc etc. I stopped the process part way, as it was throwing errors, but was none the wiser till I tried to watch a TV show that wouldn't play, after reasearching I had deleted around 1TB of TV! :(...
Here comes extundelete to the rescue.
I didn't do this in order, but what I should have done, *and did, but not in this order :P*.
umount raid, then remove nfs-server/smb server, then remove fstab entry to mount at boot.
I then turned off server, added a new HD and presented it to that VM and formatted it ext4.
I read all about compiling extundelete, but this was pretty basic stuff, so even tho the repo might be outdated, i just did sudo apt-get install -y extundelete.
I tried using --recover-directory, but it wouldn't work and ended up using the following commands.
NOTE the Symlink. I find it a GLARING oversite that extundelete doesn't let you specify the restore location. It will always restore to whatever current directory your in /RECOVERED_FILES. But, good old linux, just threw a symlink to the mounted new HD, and ran the program. Its been under 15min and I've recovered 120GB. Last time I did a file recovery it took DAYS (via USB though, this time i ripped the enclosure apart and just put the drive directly to a spare SATA), so this should be great, my only concern is I have less space in my recovery drive then in the RAID. Its like 14TB, I don't exactly have that lying around, but if it fills up, great, that will be most of my files I need and I can go from there.
Lesson : RAID doesn't = BACKUP :)... you'd think I'd learn as I got RAID last time I had a disk failure, but I still need an external USB enclosure for the hard2replace data and/or just large data pools like media.