This is definitely a real issue - increasingly more applications are migrating to being run over large distributed commodity clusters, and enabling them to share the same cluster despite operating on different frameworks is an important task that should benefit data sharing and utilization.
The main idea is to have Mesos operate as a thin resource management layer underneath the existing frameworks, providing fine-grained resource offers to frameworks, which encapsulate a certain amount of resources on the cluster. A framework can then decide which resources specifically to accept, enabling the framework to enforce e.g. data locality.
One reason this is different is that previously, the type of applications run on commodity clusters were relatively restricted, and so the Hadoop/MapReduce model was sufficient as a framework. However, as more applications are reaching the scale of needing to run in a distributed manner, the MapReduce paradigm's limits have been reached and other frameworks are necessary to compete, and running these together on the same cluster is a relatively new issue.
One hard trade-off is the focus on the simplicity of Mesos. This was done to make it very adaptable and scalable, but comes with a few downsides. One example is having Mesos offer resources to the framework and allowing the framework to reject rather than having the framework request specific resource requirements; this allows more extensibility in terms of flexible resource requirements, but means that a framework may need to wait a long time. Also, Mesos still has the frameworks make lots of scheduling decisions rather than having Mesos be a complete global scheduler; again, this means frameworks can do more complex scheduling with Mesos needing to become complex, but means that utilization may be slightly lower than optimal.
I am not sure if this paper will be particularly influential in 10 years. I think cluster resource sharing will become an increasingly more important problem, and Mesos definitely seems to be an excellent implementation of this, but the ideas presented seem relatively simple, with the main concepts being resource offers and a very thin client.
No comments:
Post a Comment