In this paper, we propose a new method to compose physically-based character controllers in low dimensional latent space. Source controllers are created by gradually updating the task parameter such as the external force applied to the body. During the optimization, instead of only saving the optimal controllers, we also keep a large number of non-optimal controllers. These controllers provide knowledge about the stable area in the controller space, and are then used as samples to construct a low dimensional manifold that represents stable controllers. During runtime, we interpolate controllers in the low dimensional space and create stable controllers to cope with the irregular external forces. Our method is best to be applied for real-time applications such as computer games.