Physically-based cloud simulation is an effective approach for synthesizing realistic cloud. However, generating clouds with desired shapes requires a time-consuming process for selecting the appropriate simulation parameters. This paper addresses such a problem by solving an inverse cloud forming problem. We propose a convolutional neural network, which has the ability of solving nonlinear optimization problems, to estimate the spatiotemporal simulation parameters for given cloud images. The cloud formation process is then simulated by using computational fluid dynamics with these control parameters as initial states. The proposed parameter-predicting model consists of three components, including the feature extraction network, the adversarial network and the parameter generation network. These subnetworks form two parallel branches for different functionality-feature extraction and parameter estimation. To solve the challenge of estimating high-dimensional spatiotemporal simulation parameters, we adapt an encoder and decoder network to compress these parameters into a low-dimensional latent space. We train the proposed deep learning model with pairwise data of time series parameters and the corresponding synthetic images, which are rendered by the density fields of the synthesized clouds under different illuminations. In the practice, our method can simulate physically plausible cloud evolution processes and generate clouds with desired shapes for the real-world and synthetic images.