[ale] School Project to Create Distributed Filesystem

[Omar Chanouha]

Okay, so I would have a cluster of servers that all have iSCSI
servers. I use linux-ha to dedicate a master server. The master then
mounts all the iSCSI drives via an mdraid, then shares the drive via
nfs. When a client connects, it gets routed to the Master over nfs and
gets all its files from it. The heartbeat is also in charge of
maintaining an up-to-date mdraid array, and reasigning a master if
neccesary.

Is that correct? If so, it seems as though there is now a funnel at
the master. The master now has to accept and serve all incoming
connections. With this algorithm, we could not have two or more
different servers accept incoming connections at the same time?

Just trying to be clear before I present this to my professor.

-OFosho, Miami Dolphin and Open Source Aficionado

On Wed, Feb 25, 2009 at 3:56 PM, Greg Freemyer <greg.freemyer at gmail.com> wrote:
> Very close, but you use heartbeat / linux-ha to run (mdraid, the
> filesystem and nfx export) on only one server at a time.  Otherwise
> you could have data consistency issues between the nodes.  That is why
> true cluster / distributed filesystems are a killer to write.
>
> heartbeat is effectively a cluster controller.
>
> For instance, you could configure it to know about 8 nodes and that it
> needs at least 6 of the nodes available for there to an operational
> cluster.  (Raid -6 is operation even if 2 nodes have failed.)
>
> Then you have a 9th IP that is called the cluster IP that floats
> between the nodes.  All of the clients connect exclusively to the
> cluster IP.
>
> Once the cluster is formed (6 or more nodes), heartbeat controls which
> of the nodes is the master.
>
> On the master it will use IP aliasing to respond to any and all tcp/ip
> connections to the cluster IP.  Further heartbeat you will need to
> configure to launch mdraid, form the raid array, mount the filesystem,
> and export it via nfs.
>
> If you are familiar with init scripts (/etc/init/*) then you are
> familiar with the scripts heartbeat uses.   Heartbeat is actually a
> superset of those scripts in that heartbeat scripts also support a
> "monitor" command.  So you just have heartbeat invoke:
>
> rcmdraid start
> rcfilesystem start
> rcnfs start
>
> on the master, and on all the slaves it calls the above with stop.
>
> Fortunately the vast majority of this already exists, but I'm not
> aware of anyone putting all the pieces together in this way.
>
> FYI: Heartbeat also provides a reliable comm channel if you need it.
> I don't think you will, but you might.
>
> Greg
>
> On Wed, Feb 25, 2009 at 3:09 PM, Omar Chanouha <ofosho at gatech.edu> wrote:
>> Greg/All, Let me get one more shot at the belt.
>>
>> Have a cluster of servers each sharing their files via iSCSI. Each
>> server then keeps an array of all the other servers via mdraid.
>> Linux-HA is used on the servers to maintain an up-to-date array of
>> iSCSI drives. Each server shares the combined RAID via an nfs server.
>> When a client wants to connect, all it needs to do is connect to one
>> of the servers over nfs and it will have all the files.
>>
>> As you said, the robustness comes into play with the mdraid update
>> scripts/making nfs work properly.
>>
>> Is that right?
>>
>> I will have to ask the teacher if he will allow that. He may not.
>>
>> Thanks,
>>
>> -OFosho, Miami Dolphin and Open Source Aficionado
>>
>> On Wed, Feb 25, 2009 at 7:26 AM, Jim Kinney <jim.kinney at gmail.com> wrote:
>>> 2009/2/24 Ken Ratliff <forsaken at targaryen.us>:
>>>>
>>>> On Feb 24, 2009, at 8:25 PM, Brian Pitts wrote:
>>>>
>>>> Another one worth checking out is gluster. It uses FUSE.
>>>>
>>>> http://www.gluster.org/
>>>>
>>>>
>>>> Gluster is.... finicky. We have it deployed in a few production clusters and
>>>> it works well when it's working, but when it breaks, it tends to break hard.
>>>
>>> Sounds like a good project for students to work on :-) It needs help.
>>>>
>>>>
>>>> _______________________________________________
>>>> Ale mailing list
>>>> Ale at ale.org
>>>> http://mail.ale.org/mailman/listinfo/ale
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --
>>> James P. Kinney III
>>> _______________________________________________
>>> Ale mailing list
>>> Ale at ale.org
>>> http://mail.ale.org/mailman/listinfo/ale
>>>
>> _______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> http://mail.ale.org/mailman/listinfo/ale
>>
>
>
>
> --
> Greg Freemyer
> Litigation Triage Solutions Specialist
> http://www.linkedin.com/in/gregfreemyer
> First 99 Days Litigation White Paper -
> http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf
>
> The Norcross Group
> The Intersection of Evidence & Technology
> http://www.norcrossgroup.com
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> http://mail.ale.org/mailman/listinfo/ale
>