The ultimate goal of machines is to help humans to solve problems.Such problems range between two extremes: structured problems for which the solution is totally defined (and thus are easily programmed by humans), and random problems for which the solution is completely undefined (and thus cannot be programmed). Problems in the vast middle ground have solutions that cannot be well defined and are, thus, inherently hard to program. Machine Learning is the way to handle this vast middle ground, so that many tedious and difficult hand-coding tasks would be replaced by automatic learning methods. There are several machine learning tasks, and this work is focused on a major one, which is known as classification. Some classification problems are hard to solve, but we show that they can be decomposed into much simpler sub-problems. We also show that independently solving these sub-problems by taking into account their particular demands, often leads to improved classification performance.