Verifying the files are there in your backup is only, like 10% of verifying that it’s a real, usable backup.
The important question is: can you successfully restore those files from the backup? Can you successfully put them back where they’re supposed to be after losing your primary copy?
A) Set up your backup/restore procedures immediately after setting up your fresh new system. And then immediately test them to see if you can successfully restore, before you’ve done anything important on the new system that you can’t afford to lose. If the restoration completely fails, no biggie. You just have to start over on setting up your fresh new system.
B) Attempt to restore your backup to a different system, not your primary one. You’ll need a second set of hardware to do that, but if you’ve got the hardware lying around, it’s a great way to test your restore procedure. If you’re upgrading your hardware anyway, it could be a good time to do this test – use your backup restoration procedure to move your data to the new hardware. (As an extra bonus, this doesn’t require any downtime on the primary system.)
C) Simulate a complete hard drive failure and replacement by replacing your primary system’s drive(s) with a blank new one. If the backup restoration fails, you should (fingers crossed) be able to just plug the old hard drive back in and everything will go back to how it was before your test.
D) Have multiple backups and multiple restore plans, and just hope to fuck that at least one of them actually works during your testing.
Option A can only be done if you’re proactive about it and do it at the right time.
Options B and C require extra hardware, but are probably the best choice if you have the hardware or can afford it.
And Option D will always have at least a tiny amount of risk associated with it.
I specifically stated that I verify the file content via md5 hash. And I keep original directory structure, so yes if i need to restore these I can.
Edit: no idea what there is to downvote here. It was definitely weird to have an md5 checking script that took days to develop and confirm working as expected and which takes days to run on several TBs of files, dismissed as simply “checking that the files are there”. No, it checks that the bytes present in the backed up copy match those in the original. You know, what an md5 checksum is.
How could I possibly be sure of that? Obviously I’ve tried my best to backup everything I would ever need. For many reasons I cannot backup every single file so I’ve made the best decisions around that I know how.
Are they restored with the right permissions and metadata automatically?
Nothing about it is automatic, by design. Doesn’t need to be. And permissions aren’t something I particularly care about since there aren’t multiple users.
I’m backing up and verifying ~ 14 TB of files and have taken great pains to ensure I’m doing everything right.
Any idea why I got downvoted? Also…why the quizzing?
My home files are not business critical infrastructure. I’m taking several steps further than any normie would take. Keeping two backups locally, confirming their byte content, sending a subset of the files to a cloud service.
To read your comments here it seems you think I’m extremely cavalier and reckless… Because I haven’t recopied 14 TB back to their original locations to ensure that… What, copying the files works? Reading the full contents of each file and comparing to the original is somehow lacking? I don’t have 14 TB in additional storage lying around to test that… Copying is a still a thing?
It’s not like I lose a billion dollars if I lose some photos. Which again, I’ve gone to great lengths to keep safe.
Are you being willfully ignorant or obstinate? Or do you not understand the concept that even with the data there, restoration of that data can fail in many ways?
A couple of times I needed to restore sites from backup, it failed. Not because the data wasnt there. Heh
Having the data is useless when the restoration process fails, which it can do due numerous reasons.
No one has explained why proving the data can be read end to end and matches the original is somehow lacking. Including you.
Probably because it isn’t lacking. For a home user who doesn’t want to lose their files, this is more than sufficient. Especially given that I have two local backups and a cloud one. None of which is exactly cheap.
Yes computers fail in many ways. What exactly are you people trying to accomplish here? Just give me anxiety? Do you have 14 TB of free storage space to lend me that I can use to do the full process of re-copying the backed up data to? …
Are you… just talking about stuff like pictures and videos and important documents? I mean, I would have thought the context was clear that that’s not really what’s being discussed. But if not, then sure, if you just have files backed up, then all you need to worry about is making sure you have enough copies of that as you need to not lose it.
Hmm. I’d better explain that.
Anywhere you have data that exists in one place, it is a matter of time before it dies. Who knows how long it’ll be, but it will eventually die.
If you have data in two places, then when it dies in one of those places, as long as it also hasn’t died in the other place, you have one copy and it will eventually die unless you replicate it somewhere else.
And many people find that when they go to read those burnded discs or read that backup external drive - oops, it’s damaged or dead. And then that data is gone.
So for unimportant things, a single backup somewhere is probably fine. But is that backup in your house with your computer that it’s also on? If your house burns, those two places are gone and your data is gone. Is that worth the tiny risk? Up to you. You know how much yo ucare about your data.
If you really want to make sure something valuable like important documents and family pictures, then ideally you want at least one copy offsite. If it’s important, it’s no bad thing to have two copies of it offsite along with perhaps one backup locally so it’s convenient. While you don’t need ten copies of data, it’s surprising how quickly 1-2 copies can go bad at the same time, or one goes bad and you don’t replace it and another goes bad and… quickly you run the risk of data loss.
For a home user who doesn’t want to lose their files,
That’s not the topic at hand, which one might’ve been able to tell from context clues.
two local backups and a cloud one.
That is a pretty good minimal setup. Not disparaging, that’s better than probably 95% if not more like 99% of people do.
Just give me anxiety?
No, you’re the one in a conversation that’s really not about your type of situation.
We’re talking about businesses who have servers - internet servers, internal servers. These run software. They have databases with largre amounts of data. They have programs that have lots of settings, configured in various ways. Servers set up to run services on the LAN and/or WAN and/or across the internet.
On your home computer, you can reinstall Windows, install Office, install Adobe, all the other software you use. And you can take the annoying time to re-customize everything to get it set up to your liking. Then copy all your documents over. You won’t have everything ready-to-go unless you use a fancy backup and restore method (that starts to touch on the subject being discussed here - that restore is not guaranteed unless you’ve tested it. It’s amazing how often that goes wrong), but it’s okay, you have time.
In a corporate environment, if something breaks and you need to restore that data and software, you need it up and running ASAP.
Now, you’d think it would be as simple as getting the hardware, installing the OS, installing the software, and restoring the data - but that’s not necessarily the case. Not the same version of the software? Data formatting might’ve changed. Settings might’ve changed. Does every version of everything work together? Underlying pieces f the system are different? Might cause things to break.
I won’t get into the technical details beyond that, but the point is that we’re not talking about just some pics and docs.
So that’s th egenesis of the misunderstandings here. It’s a wholly different topic than what you’re dealing with.
But yeah, for you, you’ve got a good backup system going. I personally have two different cloud providers for the data I want to keep the most, but that’s not all the pics and such, just for a subset of it.
I mean, I would have thought the context was clear that that’s not really what’s being discussed. But if not, then sure, if you just have files backed up, then all you need to worry about is making sure you have enough copies of that as you need to not lose it.
I mean… you’re the one changing the context. they literally were stating what they were doing with their backup, and you moved the goalposts. not even the original meme image implied it was about whatever the hell you’re on about.
No, you’re the one in a conversation that’s really not about your type of situation.
We’re talking about businesses who have servers - internet servers, internal servers
ok so you just started your own conversation without telling anyone then. yeah now it’s clear why you’re arguing with them like that. No one said anything of the sort until you came along. Their comment was a top level comment on a meme image, that has nothing to do with business requirements or servers. You’re just making up the conversation and pretending like TrickDacy was “in a conversation that’s not really about [their] type of situation”.
You are a really confusing person. The context here is that I implemented improvements to my backup process that includes verifying the content of my backups.
Then I explicitly restated that and added that I also have two copies locally and an off-site copy. So why did you think you needed to type 7 paragraphs explaining to me why I should do… All the things I said I am doing? Are you truly worried I spent several weekends of my life to create a redundant backup system, but I’m doing it for the wrong reasons? I have demonstrated that I understand all of this. It’s pretty weird that you (and a couple others) slide into the thread to insist I haven’t considered everything, all of you insisting that unless I explicitly state otherwise, whatever system I have should be treated as though huge consequences are likely if even a slight imperfection in the system exists. I would honestly be surprised if even most medium sized companies have taken as much care in implementing a backup system as I have lol.
Edit: I see, it’s a hashing algorithm. I tried a few different algorithms and had some issues with large files, and didn’t really see performance increases over md5, since it seems reading the data from disk was the bottleneck. I didn’t try this one though. I wonder if it would actually be faster.
These kinds of relentless posts finally got me to write a script that verifies all my backed up files using md5 checksums.
Verifying the files are there in your backup is only, like 10% of verifying that it’s a real, usable backup.
The important question is: can you successfully restore those files from the backup? Can you successfully put them back where they’re supposed to be after losing your primary copy?
How do you test this without risking your primary copy
That’s the tricky part, innit?
A few good options:
A) Set up your backup/restore procedures immediately after setting up your fresh new system. And then immediately test them to see if you can successfully restore, before you’ve done anything important on the new system that you can’t afford to lose. If the restoration completely fails, no biggie. You just have to start over on setting up your fresh new system.
B) Attempt to restore your backup to a different system, not your primary one. You’ll need a second set of hardware to do that, but if you’ve got the hardware lying around, it’s a great way to test your restore procedure. If you’re upgrading your hardware anyway, it could be a good time to do this test – use your backup restoration procedure to move your data to the new hardware. (As an extra bonus, this doesn’t require any downtime on the primary system.)
C) Simulate a complete hard drive failure and replacement by replacing your primary system’s drive(s) with a blank new one. If the backup restoration fails, you should (fingers crossed) be able to just plug the old hard drive back in and everything will go back to how it was before your test.
D) Have multiple backups and multiple restore plans, and just hope to fuck that at least one of them actually works during your testing.
Option A can only be done if you’re proactive about it and do it at the right time.
Options B and C require extra hardware, but are probably the best choice if you have the hardware or can afford it.
And Option D will always have at least a tiny amount of risk associated with it.
I specifically stated that I verify the file content via md5 hash. And I keep original directory structure, so yes if i need to restore these I can.
Edit: no idea what there is to downvote here. It was definitely weird to have an md5 checking script that took days to develop and confirm working as expected and which takes days to run on several TBs of files, dismissed as simply “checking that the files are there”. No, it checks that the bytes present in the backed up copy match those in the original. You know, what an md5 checksum is.
Are you sure you have all the files required? Are they restored with the right permissions and metadata automatically?
How could I possibly be sure of that? Obviously I’ve tried my best to backup everything I would ever need. For many reasons I cannot backup every single file so I’ve made the best decisions around that I know how.
Nothing about it is automatic, by design. Doesn’t need to be. And permissions aren’t something I particularly care about since there aren’t multiple users.
I’m backing up and verifying ~ 14 TB of files and have taken great pains to ensure I’m doing everything right.
Any idea why I got downvoted? Also…why the quizzing?
You check if you backed up everything correctly, and if the process works by restoring the backup and confirming they work.
But you do you, in the end it’s your neck on the line.
My home files are not business critical infrastructure. I’m taking several steps further than any normie would take. Keeping two backups locally, confirming their byte content, sending a subset of the files to a cloud service.
To read your comments here it seems you think I’m extremely cavalier and reckless… Because I haven’t recopied 14 TB back to their original locations to ensure that… What, copying the files works? Reading the full contents of each file and comparing to the original is somehow lacking? I don’t have 14 TB in additional storage lying around to test that… Copying is a still a thing?
It’s not like I lose a billion dollars if I lose some photos. Which again, I’ve gone to great lengths to keep safe.
Does the newly set up environment exactly match the previous? Same software versions?
And when the restoration of that data fails?
Are you being willfully ignorant or obstinate? Or do you not understand the concept that even with the data there, restoration of that data can fail in many ways?
A couple of times I needed to restore sites from backup, it failed. Not because the data wasnt there. Heh
Having the data is useless when the restoration process fails, which it can do due numerous reasons.
No one has explained why proving the data can be read end to end and matches the original is somehow lacking. Including you.
Probably because it isn’t lacking. For a home user who doesn’t want to lose their files, this is more than sufficient. Especially given that I have two local backups and a cloud one. None of which is exactly cheap.
Yes computers fail in many ways. What exactly are you people trying to accomplish here? Just give me anxiety? Do you have 14 TB of free storage space to lend me that I can use to do the full process of re-copying the backed up data to? …
Are you… just talking about stuff like pictures and videos and important documents? I mean, I would have thought the context was clear that that’s not really what’s being discussed. But if not, then sure, if you just have files backed up, then all you need to worry about is making sure you have enough copies of that as you need to not lose it.
Hmm. I’d better explain that.
Anywhere you have data that exists in one place, it is a matter of time before it dies. Who knows how long it’ll be, but it will eventually die.
If you have data in two places, then when it dies in one of those places, as long as it also hasn’t died in the other place, you have one copy and it will eventually die unless you replicate it somewhere else.
And many people find that when they go to read those burnded discs or read that backup external drive - oops, it’s damaged or dead. And then that data is gone.
So for unimportant things, a single backup somewhere is probably fine. But is that backup in your house with your computer that it’s also on? If your house burns, those two places are gone and your data is gone. Is that worth the tiny risk? Up to you. You know how much yo ucare about your data.
If you really want to make sure something valuable like important documents and family pictures, then ideally you want at least one copy offsite. If it’s important, it’s no bad thing to have two copies of it offsite along with perhaps one backup locally so it’s convenient. While you don’t need ten copies of data, it’s surprising how quickly 1-2 copies can go bad at the same time, or one goes bad and you don’t replace it and another goes bad and… quickly you run the risk of data loss.
That’s not the topic at hand, which one might’ve been able to tell from context clues.
That is a pretty good minimal setup. Not disparaging, that’s better than probably 95% if not more like 99% of people do.
No, you’re the one in a conversation that’s really not about your type of situation.
We’re talking about businesses who have servers - internet servers, internal servers. These run software. They have databases with largre amounts of data. They have programs that have lots of settings, configured in various ways. Servers set up to run services on the LAN and/or WAN and/or across the internet.
On your home computer, you can reinstall Windows, install Office, install Adobe, all the other software you use. And you can take the annoying time to re-customize everything to get it set up to your liking. Then copy all your documents over. You won’t have everything ready-to-go unless you use a fancy backup and restore method (that starts to touch on the subject being discussed here - that restore is not guaranteed unless you’ve tested it. It’s amazing how often that goes wrong), but it’s okay, you have time.
In a corporate environment, if something breaks and you need to restore that data and software, you need it up and running ASAP.
Now, you’d think it would be as simple as getting the hardware, installing the OS, installing the software, and restoring the data - but that’s not necessarily the case. Not the same version of the software? Data formatting might’ve changed. Settings might’ve changed. Does every version of everything work together? Underlying pieces f the system are different? Might cause things to break.
I won’t get into the technical details beyond that, but the point is that we’re not talking about just some pics and docs.
So that’s th egenesis of the misunderstandings here. It’s a wholly different topic than what you’re dealing with.
But yeah, for you, you’ve got a good backup system going. I personally have two different cloud providers for the data I want to keep the most, but that’s not all the pics and such, just for a subset of it.
I mean… you’re the one changing the context. they literally were stating what they were doing with their backup, and you moved the goalposts. not even the original meme image implied it was about whatever the hell you’re on about.
ok so you just started your own conversation without telling anyone then. yeah now it’s clear why you’re arguing with them like that. No one said anything of the sort until you came along. Their comment was a top level comment on a meme image, that has nothing to do with business requirements or servers. You’re just making up the conversation and pretending like TrickDacy was “in a conversation that’s not really about [their] type of situation”.
You are a really confusing person. The context here is that I implemented improvements to my backup process that includes verifying the content of my backups.
Then I explicitly restated that and added that I also have two copies locally and an off-site copy. So why did you think you needed to type 7 paragraphs explaining to me why I should do… All the things I said I am doing? Are you truly worried I spent several weekends of my life to create a redundant backup system, but I’m doing it for the wrong reasons? I have demonstrated that I understand all of this. It’s pretty weird that you (and a couple others) slide into the thread to insist I haven’t considered everything, all of you insisting that unless I explicitly state otherwise, whatever system I have should be treated as though huge consequences are likely if even a slight imperfection in the system exists. I would honestly be surprised if even most medium sized companies have taken as much care in implementing a backup system as I have lol.
Backup fire drill tomorrow!
Blake3(blak3?) is where it’s at
?
Edit: I see, it’s a hashing algorithm. I tried a few different algorithms and had some issues with large files, and didn’t really see performance increases over md5, since it seems reading the data from disk was the bottleneck. I didn’t try this one though. I wonder if it would actually be faster.