Backups

After several tests, now it’s time to summarize the status of the backup policy and to ask the @staff members to contribute the final piece.

As requested and agreed in Slack from the problems we had to put a site back online after Zulip’s group was shut down, we were trying to make sure it doesn’t happen again.

The strategy basically comes down to multiple backups and copies with differentiated access (but avoiding the typical complications that we network administrators tend to face and cause :smiley:). It aims to protect us from technological failure, human failure and human attack/boicot. In our case consist of:

  • AUTOMATIC daily and full backup of discuss.actualism.online. Seven in total (one for each day of the week) are maintained simultaneously.

  • At first they were stored in this server (where Discourse is installed). Now are sent automatically to Amazon S3 (a cloud space as there are so many, but for which Discourse is already preconfigured). Here they are:

image

  • AUTOMATIC sync between Amazon S3 and cloud personal spaces belonging to each of the @staff members (who will NOT have access to the locations of the other members).

I recommend using MultCloud (https://www.multcloud.com), which offers for free 30GB of data traffic per month and performs this daily synchronization between Amazon S3 and dozens of cloud services of your choice internally, from cloud to cloud, without installing any software on your machines. For now, the 7 backups total only 128MB, so monthly they don’t even reach 4GB of transfer.

Here you can see my account, which synchronizes the Actualist backup on Amazon S3 to two Mega accounts, one Dropbox account and two different locations of the same Google Drive account. I scheduled them 15 minutes apart (the synchronizations are fast). It is advisable to sync it like this, to more than one cloud service, because you can see in the image that the synchronization with Google Drive failed -it’s experiencing problems-).

image

This scheme has an additional advantage: it will copy, without having to configure anything else later, any additional element that we may include in the Amazon S3 space (for example, the backup that we will soon start to make of actualism.online -which we still don’t bother about, because for now it is only a web page in Wordpress.com-; a plugin that a third party may develop for us; the JSON with the history of Slack; etc.).

So, if @claudiu or I were suddenly floored by that mythical bus :smiley:, or if our selfs unsuspectedly “sridharized” :grinning_face_with_smiling_eyes: and we deleted the forum, any staff member could almost immediately rebuild it on another DigitalOcean droplet or any other provider. But if we also deleted the content from Amazon S3 so that no one could use those backups to recover the forum, every staff member would have them in places that neither @claudiu or I would have access to.

Aditionally, as I said, we will be able to recover even from failures or errors of (or attacks against) DigitalOcean and Amazon S3.

I suppose this should be improved by having multiple owners, if possible, of the actualism.online domain (includes the forum subdomain) or by belonging to an organization like the one @Alanji proposed.

In a post addressed only to the staff (to avoid torturing even more the rest of the members :blush:) I’ll show how to create and configure those personal accounts in MultCloud.

2 Likes

After almost 12 months of using AWS Free Tier for our backups we received the news that it is about to expire:

image

I remind you what we use AWS for: we automatically send to AWS the daily backups generated on this forum (to have also a copy of them outside the server where the forum is hosted); in turn, from AWS they are automatically sent to cloud spaces of some individual users (those who followed the instructions I gave) to avoid that only the administrators of the AWS account can access the copies in case of need to restore the forum AND to render useless a boycott that would include deleting the AWS backups as well).

I can’t find a way in AWS to forecast how much it would cost to maintain the paid srvice given our traffic/usage of it. Maybe @claudiu or someone else can.

Maybe someone can propose a similar service for free BUT it has to be S3 compatible.

Other paid S3 services, perhaps cheaper than AWS, would be (according to the creators of Discourse) “DigitalOcean Spaces” and “Minio S3”.

In any case, a year ago I used my credit card to sign up for an AWS account to try the free service for 1 year. I must now unlink it, even if we decide to continue with AWS (for example, using Alan Izatt funds. Having alternatives I would not leave an Argentina credit card associated to the regular payment of this service. Not only because of the potential problems between the card and AWS that could arise from the well known Argentinean political-financial problems, but also because of the problems that could arise in transferring me the money (to Argentina) to pay my card.

We must find another way well before August 31.

According to the AWS Calculator, if each month we were to have…

  • 10 GB of storage total
  • 150 requests to PUT data
  • 150 requests to GET data
  • download 10 GB

Then the total cost per month would be … … …

image

So I say we continue with S3 :slight_smile:

This seems a good use of the Alan Izatt Actual Freedom Fund. If you share the login info with me (perhaps privately) I can add a debit card onto the account. Or maybe better if we make an official “AIAFF” AWS account?

I’d also like to move the DigitalOcean server that hosts this site, to the AIAFF, but there was no way to move a droplet from one account to another, except for taking a snapshot, downloading it, then restoring it on the other account. The IP would change also so this’d require some reconfiguration. The other option is leave it on the current DigitalOcean account (which is actually owned by my business) and pay with the AIAFF (the only DigitalOcean stuff on it is related to this site), it’s less ‘clean’ but less work also.

2 Likes

Good. I at least had seen that our traffic/use was low.

Yes, I think will be better if I close the current AWS account (after terminating the use of all resources, as adviced) and you make an official “AIAFF” AWS account.

But I’ll wait a bit until/in case others give their opinion on the subject.

1 Like

Ok! In the meantime I already made the AIAFF AWS organization. Let me know when you’re ready to proceed, what I should do. I think I can make you a user on it so you can login on that account and set it up.

Guys, sorry for being cynical, but has anyone tried to restore from one of these backups?
I know discourse is a commercial product and this is built in functionality, but I’ve heard enough horror stories about nonfunctioning backups.

No. As a matter of fact, the ideal backup policy would be to restore a backup now and then on a clean /new droplet to verify everything works as intended. Such tests would require having another droplet available.

What is needed to setup another droplet?

I make the DO account , give someone access , they create the droplet, backup the old one , restore to the new one, then I change the DNS on the domain.

Ok, well if I can help in anyway let me know.

So in few days I’ll terminate the resources and close the account.

If you can do any of those things then I’ll set you up with access!

Ok I’ll make you a user on the AIAFFund Amazon account.

1 Like

I mean I am more use to Azure and Microsoft tech stack but I am sure I can figure out. Have a few friends who work in infrastructure too, so if I get stuck have a lot of people I can lean on for advice.

Cool! I’m out camping so have less computer time but I’ll get the accounts set up and give you access when I get a chance. Thanks for the offer :man_dancing:

@Miguel @son_of_bob i’ve invited you both to the old digitalocean account (currently hosting the site) and the new one. so you guys should have all you need to move the droplet to the new one (that’s funded by the AIAIFF).

Some walkthrough on how to transfer it: How do I transfer a Droplet to another user? :: DigitalOcean Documentation

I think only missing piece will be for me to change the DNS settings once the new IP is ready, I can do it once new site is ready to go. Can put an alert up until I change the IP that messages will be lost until the move. I am generally around this time in the evenings and can do the switch around this hours.

1 Like

Thanks @claudiu I have been able to access both without an issue.

Are you happy for me to proceed then?

1 Like