We have taken over an application which had "pretend" failover. Essentially it had two servers with automatic failover between the two. However they both relied on another server for providing the data storage via NFS all the way down to the MySQL server. Interesting how one can manage to provide no real failover with 3 servers. Obviously we want to fix this and actually a number of servers have already been bought. So now we have moved the MySQL server with a cold standby on separate machines. We also have 3 frontend servers which we want to load balance. We will probably use memcache to manage the sessions, as we are not so worried about a lost session when a crash occurs. But now comes the problem: There are a lot of places where administrators and end users can upload files, which end up in the file system. Now how do we replicate those files across all the nodes?

The temporary approach we will be taking will be using NFS with rsync. One server will be the master for the NFS for the other servers. Behind the scenes we will use rsync to replicate the relevant folders to the other servers for failover. Obviously this means that the other servers need to go to the master for writes and reads. Also the rsync is not going to be very real time at all. However in a crash scenario we will only need to deal with a few seconds worth if lost files that should be recoverable manually, while one of the other server takes over the NFS serving in the mean time. Before readding the server we will need to resync it. Not exactly beautiful but it will get the job done.

Now what to do in the future? Moving all the writes to the database and then caching into the local file system is probably going to require way too much effort to do now. What we really need is a clustered file system that would keep all reads local but broadcast all writes in real time. I looked at DRDB but it seems to be limited to 2 servers (though there are hacks available to expand that number). I also looked at GFS2, but after reading tons of FAQ's I am left with more questions than answers. So is there anything out there?

It seems like most clustering products don't do the read local part of your requirement, because they allow (or assume) the storage nodes to be completely separate from your client nodes (client in the sense of requesting a file from the cluster). That said, you could try looking at MogileFS and set the # of replicas to the number of servers you have. I assume that'd put a copy of each file on every server. I imagine the read local requirement could be met with some tweaks here or there.

Why not use a local proxy server with Amazon's S3 as the write store?

Wordpress.com just moved to this model and we at Lookery.com are moving this way as well.

http://blog.apokalyptik.com/2007/10/10/so-you-wanna-see-an-image/

They're also moving Gravatar over to the same system:
http://barry.wordpress.com/2007/10/26/making-gravatar-fast-again/

At one point, we'd cobbled together a (slightly more elegant than straight rsync) system using inotify as a trigger for the sync.

http://inotify-tools.sourceforge.net/#info

I've been wanting to figure this out as well.

I tried to figure out how to make my webservers read-only, and push file uploads just to the NFS server...

I've thought about Lustre and Coda/AFS.

I've also thought about GFS2, OCFS2 - but those require shared storage. I could buy iSCSI-exported space, but that seems a bit complex at the moment to jump into.

I used DRBD to mirror two servers in the past. It was pretty neat, and I'm sure has come an even longer way now, but it is limited to only two servers and does require block-level partitioning and planning...

MogileFS seems like a good contender - as long as you can build it into your applications. I run a shared hosting cluster, I'd have to "enable" all my client's apps to work with MogileFS... not very fun.

Doing these pseudo-CDN methods of storing content somewhere as the source site and then proxying/grabbing it local on demand has been an idea I thought was neat and wanted to look into more. However, each of my webservers has 220ish gigs available (x3) and I have more content then that already, so I'd have to be expiring things at some point (an LRU idea like memcache) ... this is very cool though you've got some good ideas/responses, I'll be sure to keep checking in.

I'd be interested to get more information about what "andrew" mentioned. Do you have a sample script? What is the performance like? Is there *any* delay? etc.

Thanks!

What about Unison? We've used it on load balanced setups with some success before ourselves, and it's pretty easy to set up. The advantage over rsync is it's multi-directional. Here's a basic step-by-step for setting it up (some stuff near the end is specific to our software, but 99% of the tutorial is generic):

http://cookbook.sitellite.org/index/sitewiki-app/show.LoadBalancingWithUnison

Hope that helps :)

MogileFS is not POSIX complain fs, everytime injecting and exporting files from it's own fs, can be very slow :( - correct me if I'm wrong

Unison isn't good solution for quick replication of cluster nodes. What can be used instead of Unison is Csync2, which is easy to use and fast enough for some cases. It's not realtime data replication, but with some external logic it can work pretty well. I must notify that it's a hell to compile this soft on CentOS 5, becaouse of dependencies..This howto can save a lot of time http://zhenhuiliang.blogspot.com/

Does anyone have some expirences with Lustre? It doesn't support data replication itself, but with DRBD it can work - but I'm affraid that only for 2 nodes.

I checked this solutions (some just from documentations, someones are already tested):

I can't recomment these - V9FS, Gfarm,PVFS, NBD, MogileFS, Intermezzo, Unison
This ones looks better - DRBD, Csync2, Lustre, GFS, NFS4,

Yeah Lustre and GFS (both with DRBD) can be hard to implement.

DRBD itself doesn't support RW on both nodes.

How about MogileFS?

I'm looking for a solution for a similar reason: I want to keep a folder of two or three machines in my local network synced. I just found this, maybe it is another inspiration!?

http://files.calum.org/network-raid.html

If you don't find exactly what you need (or perhaps for backup purposes), you could use LVM/ZFS snapshots, capturing the incremental difference, and then applying that to the appropriate places......