Realtime Simulation of ThinShell Deformable Materials using CNNBased Mesh Embedding
Abstract
We address the problem of accelerating thinshell deformable object simulations by dimension reduction. We present a new algorithm to embed a highdimensional configuration space of deformable objects in a lowdimensional feature space, where the configurations of objects and feature points have approximate onetoone mapping. Our key technique is a graphbased convolutional neural network (CNN) defined on meshes with arbitrary topologies and a new mesh embedding approach based on physicsinspired loss term. We have applied our approach to accelerate highresolution thin shell simulations corresponding to clothlike materials, where the configuration space has tens of thousands of degrees of freedom. We show that our physicsinspired embedding approach leads to higher accuracy compared with prior mesh embedding methods. Finally, we show that the temporal evolution of the mesh in the feature space can also be learned using a recurrent neural network (RNN) leading to fully learnable physics simulators. After training our learned simulator runs faster and the accuracy is high enough for robot manipulation tasks.
FigFigure LABEL:#1 \newrefformatfigFigure LABEL:#1 \newrefformatparSection LABEL:#1 \newrefformatappenAppendix LABEL:#1 \newrefformatsecSection LABEL:#1 \newrefformatsubSection LABEL:#1 \newrefformattableTable LABEL:#1 \newrefformatalgAlgorithm LABEL:#1 \newrefformatAlgAlgorithm LABEL:#1 \newrefformatDefDefinition LABEL:#1 \newrefformatThmTheorem LABEL:#1 \newrefformatstepStep LABEL:#1 \newrefformatlnLine LABEL:#1 \newrefformateqEquation LABEL:#1 \newrefformatpbProblem LABEL:#1 \newrefformatitItem LABEL:#1 \newrefformatteTerm LABEL:#1 \newrefformatEqEquation (LABEL:eq:#1)
I Introduction
A key component in robot manipulation tasks is a dynamic model of target objects to be manipulated. Typical applications include cloth manipulation [26, 31], liquid manipulation [42], and inhand rigid object manipulation [44]. Of these objects, cloth is unique in that it is modeled as a thinshell, i.e., a 2D deformable object embedded in a 3D workspace. To model the dynamic behaviors of thinshell deformable objects, people typically use highresolution meshes (e.g. with thousands of vertices) to represent the deformable objects. Many techniques have been developed to derive a dynamic model under a meshbased representation, including the finiteelement method [30], the massspring system [6, 11], the thinshell model [20], etc. However, the complexity of these techniques can vary from to [19], where is the number of DOFs, which makes them very computationally cost on highresolution meshes. For example, [38] reported an average computational time of over minute for predicting a single future state of a thinshell mesh with around vertices. This simulation overhead is a major cost in various cloth manipulation algorithms including [26, 31, 29].
In order to reduce the computational cost, one recent trend is to develop machine learning methods to compute lowdimensional embeddings of these meshes. Lowdimensional embeddings were original developed for applications such as image compression [28] and dimension reduction [55]. The key idea is to find a lowdimensional feature space with approximate onetoone mapping between a lowdimensional feature point and a highdimensional mesh shape. So that the lowdimensional feature point can be treated as an efficient, surrogate representation of the original mesh.
However, computing lowdimensional embeddings for general meshes poses new challenges because, unlike 2D images, meshes are represented by a set of unstructured vertices connected by edges and these vertices can undergo large distortions when cloth deforms. As a result, a central problem in representing mesh deformation data is to find an effective parameterization of the feature space that can handle arbitrary mesh topologies and large, nonlinear deformations. Several methods for lowdimensional mesh embeddings are based on PCA [2], localized PCA [39], and Gaussian Process [54]. However, these methods are based on vertexposition features and cannot handle large deformations.
Main Results: We present a novel approach that uses physicsbased constraints to improve the accuracy of lowdimensional embedding of arbitrary meshes for deformable simulation. We further present a fully learnable physics simulator of clothes in the feature space. The novel components of our algorithm include:

A graphbased CNN [15] mapping highDOF configurations to lowDOF features, which extends conventional CNN to handle arbitrary mesh topology, with loss function defined in the ACAP feature space.

A mesh embedding approach aware of the inertial and internal potential forces used by a physical simulator, which is achieved by introducing a physicsinspired loss function term, i.e., vertexlevel physicsbased loss term (PBloss), and better preserves the material properties of the meshes.

A stateful, recurrent featurespace physics simulator that predicts the temporal changes of meshes in the feature space, which is accurate enough for learning cloth features and training cloth manipulation controllers (see \prettyreffig:simulateRobot).
To test the accuracy of our method, we construct multiple datasets by running cloth simulations using a highresolution mesh under different material models, material parameters, and mesh topologies. We show that our embedding approach leads to better accuracy in terms of physics rule preservation than prior method [46] that uses only a data term, with up to improvement. We have also observed up to and improvements in mesh embedding accuracy on commonly used metrics such as and . Finally, we show that our feature space physics simulator can robustly predict dynamic behaviors of clothes undergoing unseen robot manipulations, while achieving speedup over simulators running in the highdimensional configuration space.
The paper is organized as follows. We first review related work in \prettyrefsec:related. We define our problem and introduce the basic method of lowdimensional mesh embedding in \prettyrefsec:VDE. We introduce our novel PBloss and the learnable simulation architecture in \prettyrefsec:physics. Finally, we describe the applications in \prettyrefsec:app and highlight the results in \prettyrefsec:results.
Ii Related Work and Background
We summarize related work in mesh deformations and representations, deformable object simulations, and machine learning methods for mesh deformations.
Deformable Simulation for Robotics are frequently encountered in service robots applications such as laundry cleaning [8, 29] and automatic cloth dressing [12]. Studying these objects can also benefit the design of soft robots [37, 16]. While these soft robots are usually 3D volumetric deformable objects, we focus on 2D shelllike deformable objects or clothes. In some applications such as visual servoing [25] and tracking [10], deformable objects are represented using point clouds. In other applications including modelbased control [41] and reconstruction [48], the deformable objects are represented using meshes and their dynamics are modeled by discretizing the governing equations using the finite element method (FEM). Solving the discretized governing equation is a major bottleneck in training a cloth manipulation controller, e.g., [5] reported up to 5 hours of CPU time spend on thinshell simulation which is 45 times more costly than the control algorithm.
Deformable Object Simulations is a key component in various modelbased control algorithms such as virtual surgery [3, 4, 32] and soft robot controllers [41, 14, 26]. However, physics simulators based on the finite element method [30], the boundaryelement method [9], or simplified models such as the massspring system [11] have a superlinear complexity. An analysis is given in [19], resulting in complexity, where is the number of DOFs. In a highresolution simulation, can be in the tens of thousands. As a result, learningbased methods have recently been used to accelerate physics simulations. This can be done by simulating under a lowresolution using FEM and then upsampling [51] or by learning the dynamics behaviors of clothes [40] and fluids [50]. However, these methods are either not based on meshes [50] or not able to handle arbitrary topologies [40].
Machine Learning Methods for Mesh Deformations has been in use for over two decades, of which most methods are essentially lowdimensional embedding techniques. Early work are based on principle component analysis (PCA) [2, 55, 39] that can only represent small, local deformations or Gaussian processes [49, 54] that are computationally costly to train and do not scale to large datasets. Recently, deep neural networks have been used to embed highdimensional nonlinear functions [28, 43]. However, these methods rely on regular data structures such as 2D images. To handle meshes with arbitrary topologies, earlier methods [36] represent a mesh as a 3D voxelized grid or reconstruct 3D shapes from 2D images [52] using a projection layer. Recently, methods have been proposed to define CNN directly on mesh surfaces, such as CNN on parametrized texture space [35], and CNN based on spatial filtering [15]. The later has been used in [46] to embed largescale deformations of general meshes. Our contribution is orthogonal to these techniques and can be used to improve the embedding accuracy for any one of these methods.
Iii LowDimensional Mesh Embedding
In this section, we provide an overview of lowdimensional embedding of thin shell like meshes such as clothes. Our goal is to represent a set of deformed meshes, , with each mesh represented using a set of vertices, denoted as . We denote the th vertex as . Here and . These vertices are connected by edges, so we can define the 1ring neighbor set, , and the 2ring neighbor set, , for each , as shown in \prettyreffig:method (c). Our goal is to find a map , where is a lowdimensional feature and such that, for each , there exists a where is mapped to a mesh close to . To define such a function, we use graphbased CNN and ACAP features [17] to represent largescale deformations.
Iiia ACAP Feature
For each , an ACAP feature is computed by first finding the deformation gradient on each vertex:
(1) 
where are cotangent weights [13]. Here, we use as a reference shape. Next, we perform polar decomposition to compute where is orthogonal and is symmetric. Finally, is transformed into logspace in an asconsistentaspossible manner using mixedinteger programming. The final ACAP feature is defined as due to the symmetry of . We denote the ACAP feature transform as: . It is suggested, e.g., in [23], that mapping to the ACAP feature space leads to better effectiveness in representing largescale deformations. Therefore, we define our mapping function to be and then recover via the inverse feature transform: .
IiiB GraphBased CNN for Feature Embedding
The key idea in handling arbitrary mesh topologies is to define as a graphbased CNN using local filters [15]:
where is the number of convolutional layers and is the transpose of a graphbased convolutional operator. Finally, is the transpose of a fully connected layer. Each layer is appended by a leaky ReLU activation layer. A graphbased convolutional layer is a linear operator defined as:
where are optimizable weights and biases, respectively. All the weights in the CNN are trained in a selfsupervised manner using an autoencoder and the reconstruction loss:
where is a mirrored encoder of defined as:
The construction of this CNN is illustrated in \prettyreffig:method (a). In the next section, we extend this framework to make it aware of physics rules.
Iv PhysicsBased Loss Term
We present a novel physicsinspired loss term that improves the accuracy of lowdimensional mesh embedding. Our goal is to combine physicsbased constraints with graphbased CNNs, where our physicsbased constraints take a general form and can be used with any material models such as FEM [38] and massspring system [11]. We assumes that is generated using a physics simulator that solves a continuoustime PDE of the form:
(2) 
where is the mass matrix and is the time. This form of governing equation is the basis for stateoftheart thin shell simulators including [11, 38]. models internal and external forces affecting the current mesh . The force is also a function of the current control parameters , which are the positions of the grasping points on the mesh (red dots of \prettyreffig:datasetVis). This continuous time PDE \prettyrefeq:PDE can be discretized into timesteps such that is the position of at time instance , where is the timestep size. A discrete physics simulator can determine all given the initial condition and the sequence of control parameters by the recurrent function:
(3) 
where is a discretization of \prettyrefeq:PDE. To define this discretization, we use a derivation of [33] that reformulates as the following optimization:
(4) 
Note that \prettyrefeq:PLOSS is just one possible implementation of \prettyrefeq:SIM. Here the first term models the kinematic energy, which requires each vertex to move in its own velocity as much as possible if no external forces are exerted. The second term models forces caused by various potential energies at configuration . In this work, we consider three kinds of potential energy:

Gravitational energy , where is the gravitational acceleration vector.

Stretch resistance energy, , models the potential force induced by stretching the material.

Bending resistance energy, , models the potential force induced by bending the material.
There are many ways to discretize , such as the finite element method used in [38] or the massspring model used in [33, 11]. Both formulations are evaluated in this work.

[11] models the stretch resistance term, , as a set of Hooke’s springs between each vertex and vertices in its 1ring neighbors. In addition, the bend resistance term, , is defined as another set of Hooke’s springs between each vertex and vertices in its 2ring neighbors. (\prettyreffig:method (c))

[38] models the stretch resistance term, , as a linear elastic energy resisting the inplane deformations of each mesh triangle. In addition, the bend resistance term, , is defined as a quadratic penalty term resisting the change of the dihedral angle between any pair of two neighboring triangles. (\prettyreffig:method (d))
Our approach uses \prettyrefeq:PLOSS as an additional loss function for training . Since \prettyrefeq:PLOSS is used for data generation, using it for mesh deformation embedding should improve the accuracy of the embedded shapes. However, there are two inherent difficulties in using as an loss function. First, is defined on the vertex level as a function of , not on the feature level as a function of . To address this issue, we use the inverse function to reconstruct from . The implementation of is introduced in \prettyrefsec:IssueA. By combining with , we can train the mesh deformation embedding network using the following loss:
(5) 
Our second difficulty is that the embedding network is stateless and does not account for temporal information. In other words, function only takes as input, while \prettyrefeq:PLOSS requires . To address this issue, we use a small, fully connected, recurrent network to represent the physics simulation procedure in the feature space. The training of this stateful network is introduced in \prettyrefsec:IssueB. Finally, in addition to the PBloss, we also add an autoencoder reconstruction loss on the vertex level as a regularization:
Iva The Inverse of the ACAP Feature Extractor
The inverse of the function (black block in \prettyreffig:method) involves three steps. Fortunately, each step can be easily implemented in a modern neural network toolbox such as TensorFlow [1]. The first step computes from using the Rodrigues’ rotation formula, which involves only basic mathematical functions such as dotproduct, crossproduct, and the cosine function. The second step computes from , which is a matrixmatrix product. The final step computes from . According to \prettyrefeq:RECON, this amounts to premultiplying the inverse of a fixed sparse matrix, , representing the Poisson reconstruction. However, this is rank3 deficient because it is invariant to rigid translation. Therefore, we choose to define a pseudoinverse by fixing the position of the grasping points :
(6) 
which can be prefactorized. Here is a matrix selecting the grasping points.
IvB Stateful Recurrent Neural Network
A physics simulation procedure is Markovian, i.e. current configuration only depends on previous configuration of the mesh. As a result, is a function of both , , and , which measures the violation of physical rules. However, our embedding network is stateless and only models . In order to learn the entire dynamic behavior, we augment the embedding network with a stateful, recurrent network represented as a multilayer perceptron (MLP). This MLP represents a physically correct simulation trajectory in the feature space and is also Markovian, denoted as:
(7) 
Here the additional control parameters are given to as additional information. We can build a simple reconstruction loss below to optimize :
In addition, we can also add PBloss to train this MLP, for which we define on a sequence of meshes by unrolling the recurrent network:
(8)  
However, we argue that \prettyrefeq:LPHYS will lead to a physically incorrect result and cannot be directly used for training. To see this, we note that \prettyrefeq:PLOSS is the variational form of \prettyrefeq:PDE. So that is physically correct when is at its local minima, i.e. the following partial derivative vanishes:
(9) 
However, if we sum up over a sequence of meshes and require the summedup loss to be at a local minimum, as is done in \prettyrefeq:LPHYS, then we are essentially requiring the following derivatives to vanish:
(10) 
The difference between \prettyrefeq:VIO_MEASURE and \prettyrefeq:VIO_MEASURE_SUM is the reason that \prettyrefeq:LPHYS gives an incorrect result. To resolve the problem, we slightly modify the back propagation procedure of our training process by setting the partial derivatives of with respect to its first two parameters to zero:
which, combined with \prettyrefeq:VIO_MEASURE_SUM, leads to \prettyrefeq:VIO_MEASURE. (We add similar gradient constraints when optimizing over \prettyrefeq:embedding_phys.) This procedure is equivalent to an alternating optimization procedure, where we first compute a sequence of feature space coordinates, , using the recurrent network (\prettyrefeq:MLP) and then fix the first two parameters and optimize with respect to its third parameter .
V Applications
The two novel components in our method, the operator and the stateful PBloss, enable a row of new applications, including realtime cloth inverse kinematics and feature space physics simulations.
Va Cloth Inverse Kinematics
Our first application allows a robot to grasp several points of a piece of cloth and then infer the full kinematic configuration of the cloth. Such inverse kinematics can be achieved by minimizing a highdimensional nonlinear potential energy, such as ARAP energy [45], which is computationally costly. Using the inverse of the ACAP feature extractor, our method allows vertexlevel constraints. Therefore, we can perform solve for the cloth configuration by a fast, lowdimensional minimization in the feature space as:
where we treat all the grasped vertices as control parameters used in \prettyrefeq:invACAP. This application is stateless and the user controls a single feature of a mesh, , so that we drop the kinetic term in and only retain the potential term . Some inverse kinematic examples generated using this formulation are shown in \prettyreffig:control. Note that detailed wrinkles and clothlike deformations are synthesized in unconstrained parts of the meshes.
VB Feature Space Physics Simulation
For our second application, we approximate an entire cloth simulation sequence (\prettyrefeq:SIM) in the 128dimensional feature space. Starting from , we can generate an entire sequence of frames by using the recurrent relationship in \prettyrefeq:MLP and can recover the meshes via the function . Such a latent space physics model has been previously proposed in [50] for voxelized grids, while our model works on surface meshes. We show two synthesized simulation sequences in \prettyreffig:simulate.
VC Accuracy of Learned Simulator for Robotic Cloth Manipulation
We show three benchmarks (\prettyreffig:simulateRobot) from robot cloth manipulation tasks defined in prior work [26]. In these benchmarks, the robot is collaborating with human to maintain a target shape of a piece of cloth. To design such a collaborating robot controller, we use imitation learning by teaching the robot to recognized cloth shapes under various, uncertain human movements. Our learnable simulator can be used to efficiently generate these cloth shapes for training the controller. To this end, we train our neuralnetwork using the original dataset from [26] obtained by running the FEMbased simulator [38], which takes 3 hours. During test time, we perturb the human hands’ grasp points along randomly directions. Our learned physical model can faithfully predict the dynamic movements of the cloth.
Vi Results
To evaluate our method, we create two datasets of cloth simulations using \prettyrefeq:PLOSS. Our first dataset is called SHEET, which contains animations of a squareshaped cloth sheet swinging down under different conditions, as shown in \prettyreffig:datasetVis (a). This dataset involves simulation sequences, each with frames. Among these sequences, the first sequence uses the massspring model [11] to discretize \prettyrefeq:PDE and the cloth mesh has no holes (denoted as SHEET+[11]). The second sequence uses the mass spring model and the cloth mesh has holes, as shown in \prettyreffig:simulate (a,b), which is denoted as (SHEET+[11]+holes). The third sequence uses FEM [38] to discretize \prettyrefeq:PDE and the cloth mesh has no holes (denoted as SHEET+[38]). The forth sequence uses FEM to discretize \prettyrefeq:PDE and the cloth interacts with an obstacle, as shown in \prettyreffig:datasetVis (c) (denoted as SHEET+[38]+obstacle). In the SHEET dataset, the cloth mesh without holes has vertices and the cloth mesh with holes has vertices. Our second dataset is called BALL, which contains animations of a cloth ball being dragged up and down under different conditions, as shown in \prettyreffig:datasetVis (d). This dataset also involves simulation sequences, each with frames. Using the same notation as the SHEET dataset, the sequences in the BALL dataset are (BALL+[11], BALL+[38], BALL+[38]+, BALL+[38]+). Here means that we multiply the stretch/bend resistance term by , making the material softer and less resilient when stretched or bent. In the BALL dataset, the cloth ball mesh has vertices. During comparison, for each dataset, we select first 12 frames in every 17 frames to form the training set. The other frames are used as the test set.
Via Implementation
We implement our method using Tensorflow [1] and we implement the PBloss as a special network layer. When there is an obstacle interacting with the cloth, we model the collision between the cloth and the obstacle using a special potential term proposed in [18]. For better conditioning and a more robust initial guess, our training procedure is broken into three stages. During the first stage, we use the loss:
to optimize . During the second stage, we use the loss:
to optimize . Finally, we add a finetuning step and use the loss:
to optimize both and . Notice that, in order to train the mesh embedding network and at the same time, we feed:
to for better stability during the third stage.
ViB Physics Correctness of LowDimensional Embedding
We first compare the quality of mesh deformation embeddings using two different methods. The quality of embedding is measured using three metrics. The first metric is the root mean square error, [27], which measures the averaged vertexlevel error over all shapes and vertices. Our second metric is the STED metric, [47]. This metric linearly combines several aspects of errors crucial to visual quality, including relative edge length changes and temporal smoothness. However, since is only meaningful for consecutive frames, we compute for the consecutive frames in every frames, which is the test set. Finally, we introduce a third metric, physics correctness, which measures how well the physics rule is preserved. Inspired by \prettyrefeq:VIO_MEASURE, physics correctness is measured by the norm of partial derivatives of : . Note that the absolute value of can vary case by case. For example, using the FEM method can be orders of magnitude larger than that using the massspring system in our dataset. So that only the relative value of indicates improvement in physics correctness.
Our first experiment compares the accuracy of mesh embedding with or without PBloss. The version without PBloss is our baseline, which is equivalent to adding vertex level loss to [46]. In addition, we remove the sparsity regularization from [46] to make it consistent with our formulation. We denote this baseline as [46]+. A complete summary of our experimental results is given in \prettyreftable:comparisonA. The benefit of threestage training is given in \prettyreftable:MLP. From \prettyreftable:comparisonA, we can see that including PBloss significantly and consistently improves . This improvement is large, up to on the SHEET+[38] dataset. In addition, by adding , our method also better recognizes the relationship between each model and embeds them, thus improves in all the cases. However, our method sometimes sacrifice as temporal smoothness is not modeled explicitly in our method.
ViC Discriminability of Feature Space
In our second experiment, we evaluate the discriminability of mesh embedding by classifying the meshes using their feature space coordinates. Note that our datasets (\prettyreffig:datasetVis) are generated by moving the grasping points back and forth. We use these movement directions as the labels for classification. For the SHEET dataset, we have 6 labels: , where means rotating the grasping points around axes. For the BALL dataset, we have 2 labels: . Note that it is trivial to classify the meshes if we know the velocity of the grasping points. However, this information is missing in our feature space coordinates because ACAP features are invariant to global rigid translation, which makes the classification challenging. \prettyreffig:classify shows the feature space visualization using tSNE [34] compressed to 2 dimensions. We also report retrieval performance in the KNN neighborhoods across different K’s, using method suggested by [53]. The normalized discounted cumulative gain (DCG) on the test set for SHEET+[11] is and for BALL+[11] is .
ViD Sensitivity to Training Parameters
In our third experiment, we evaluate the sensitivity of our method with respect to the weights of loss terms, as summarized in \prettyreftable:comparisonB. Our method outperforms [46]+ under a range of different parameters. We have also compared our method with other baselines such as [39] and [24]. As shown in the last two columns of \prettyreftable:comparisonB, they generate even worse result, which indicates that [46]+ is the best baseline.
ViE Robustness to Mesh Resolutions
In our final experiment, we highlight the robustness of our method to different mesh resolutions by lowering the resolution of our dataset. For SHEET+[11], we create a midresolution counterpart with vertices and a lowresolution counterpart with vertices. On these two new datasets, we compare the accuracy of mesh embedding with or without PBloss. The results are given in \prettyreftable:comparisonC. Including PBloss consistently improves and overall embedding quality, no matter the resolution used.
Dataset  Method  
SHEET+[11]  baseline  
2nd stage  
3rd stage  
SHEET+[11]+holes  baseline  
2nd stage  
3rd stage 
ViF Difficulty in Contact Handling
One exception appears in the SHEET+[38]+obstacle (blue row in \prettyreftable:comparisonA), where our method deteriorates physics correctness. This is the only dataset where the mesh is interacting with an obstacle. The deterioration is due to the additional loss term penalizing the penetration between the mesh and the obstacle. This term is nonsmooth and has very high value and gradient when the mesh is in penetration, making the training procedure unstable. This means that direct learning a feature mapping for meshes with contacts and collisions can become unstable. However, we can solve this problem using a twostage method, where we first learn a feature mapping for meshes without contacts and collisions, and then handle contacts and collisions at runtime using conventional method [21], as is done in [7].
Vii Conclusion & Limitations
In this paper, we present a new method that bridges the gap between mesh embedding and and physical simulation for efficient dynamic models of clothes. We achieve lowdimensional mesh embedding using a stateless, graphbased CNN that can handle arbitrary mesh topologies. To make the method aware of physics rules, we augment the embedding network with a stateful feature space simulator represented as a MLP. The learnable simulator is trained to minimize a physicsinspired loss term (PBloss). This loss term is formulated on the vertex level and the transformation from the ACAP feature level to the vertex level is achieved using the inverse of the ACAP feature extractor.
Our method can be used for several applications, including fast inverse kinematics of clothes and realtime feature space physics simulation. We have evaluated the accuracy and robustness of our method on two datasets of physics simulations with different material properties, mesh topologies, and collision configurations. Compared with previous models for embedding, our method achieves consistently better accuracy in terms of physics correctness and the mesh change smoothness metric ([47]).
A future research direction is to apply our method to other kinds of deformable objects, i.e., volumetric objects [22]. Each and every step of our method can be trivially extended to handle volumetric objects by replacing the triangle surface mesh with a tetrahedral volume mesh. A minor limitation of the current method is that the stateful MLP and the stateless mesh embedding cannot be trained in a fully endtoend fashion. We would like to explore new optimization methods to train the two networks in an endtoend fashion while achieving good convergence behavior.
References
 [1] (2016) TensorFlow: a system for largescale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265–283. External Links: Link Cited by: §IVA, §VIA.
 [2] (2000) Representing animations by principal components. Computer Graphics Forum 19 (3), pp. 411–418. Cited by: §I, §II.
 [3] (2008) Motion planning under uncertainty for imageguided medical needle steering. The International journal of robotics research 27 (1112), pp. 1361–1374. Cited by: §II.
 [4] (2009) Sensorless motion planning for medical needle insertion in deformable tissues. IEEE Transactions on Information Technology in Biomedicine 13 (2), pp. 217–225. Cited by: §II.
 [5] (2016) Dexterous manipulation of cloth. In Computer Graphics Forum, Vol. 35, pp. 523–532. Cited by: §II.
 [6] (1998) Large steps in cloth simulation. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’98, New York, NY, USA, pp. 43–54. External Links: ISBN 0897919998, Link, Document Cited by: §I.
 [7] (2010) Subspace selfcollision culling. ACM Trans. on Graphics (SIGGRAPH 2010) 29 (4), pp. 81:1–81:9. Cited by: §VIF.
 [8] (2011Sep.) Bimanual robotic cloth manipulation for laundry folding. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. , pp. 1413–1419. External Links: Document, ISSN 21530866 Cited by: §II.
 [9] C. A. Brebbia and M. H. Aliabadi (Eds.) (1993) Industrial applications of the boundary element method. Computational Mechanics, Inc., Billerica, MA, USA. External Links: ISBN 1853121835 Cited by: §II.
 [10] (2008) Segmentation and recognition using structure from motion point clouds. In Proceedings of the 10th European Conference on Computer Vision: Part I, ECCV ’08, Berlin, Heidelberg, pp. 44–57. External Links: ISBN 9783540886815 Cited by: §II.
 [11] (2002) Stable but responsive cloth. In Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’02, New York, NY, USA, pp. 604–611. External Links: ISBN 1581135211, Link, Document Cited by: §I, §II, 1st item, §IV, Fig. 6, §VIC, §VIE, TABLE I, TABLE II, TABLE III, TABLE IV, §VI.
 [12] (201812) Learning to dress: synthesizing human dressing motion via deep reinforcement learning. ACM Trans. Graph. 37 (6), pp. 179:1–179:10. External Links: ISSN 07300301 Cited by: §II.
 [13] (1999) Implicit fairing of irregular meshes using diffusion and curvature flow. In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’99, New York, NY, USA, pp. 317–324. External Links: ISBN 0201485605, Link, Document Cited by: §IIIA.
 [14] (2013) Control of Elastic Soft Robots based on RealTime Finite Element Method. In ICRA 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, France. Cited by: §II.
 [15] (2015) Convolutional networks on graphs for learning molecular fingerprints. In NIPS, pp. 2224–2232. Cited by: 1st item, §II, §IIIB.
 [16] (201805) Bioinspired octopus robot based on novel soft fluidic actuator. In 2018 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 1583–1588. Cited by: §II.
 [17] (2017) Sparse Data Driven Mesh Deformation. arXiv:1709.01250. Cited by: §III.
 [18] (2015) Optimization integrator for large time steps. IEEE transactions on visualization and computer graphics 21 (10), pp. 1103–1115. Cited by: §VIA.
 [19] (1988) On the complexity of sparse $qr$ and $lu$ factorization of finiteelement matrices. SIAM Journal on Scientific and Statistical Computing 9 (5), pp. 849–861. External Links: Document Cited by: §I, §II.
 [20] (2003) Discrete shells. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA ’03, AirelaVille, Switzerland, Switzerland, pp. 62–67. External Links: ISBN 1581136595 Cited by: §I.
 [21] (2000) Simulation of nonpenetrating elastic bodies using distance fields. Cited by: §VIF.
 [22] (201907) 3d deformable object manipulation using deep neural networks. IEEE Robotics and Automation Letters PP, pp. 1–1. External Links: Document Cited by: §VII.
 [23] (201107) Interactive shape interpolation through controllable dynamic deformation. IEEE Transactions on Visualization and Computer Graphics 17 (7), pp. 983–992. External Links: Document, ISSN 10772626 Cited by: §IIIA.
 [24] (2014) Sparse localized decomposition of deformation gradients. Comp. Graph. Forum 33 (7), pp. 239–248. Cited by: §VID, TABLE II.
 [25] (2018) Cloth manipulation using random forestbased controller parametrization. CoRR abs/1802.09661. Cited by: §II.
 [26] (2018) Fast motion planning for highdof robot systems using hierarchical system identification. External Links: arXiv:1809.08259 Cited by: §I, §II, Fig. 4, §VC.
 [27] (2010) Fast and efficient skinning of animated meshes. Computer Graphics Forum 29 (2), pp. 327–336. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.14678659.2009.01602.x Cited by: §VIB.
 [28] (2013) Autoencoding variational bayes.. arXiv:1312.6114 . Cited by: §I, §II.
 [29] (2013) A constraintaware motion planning algorithm for robotic folding of clothes. In Experimental Robotics, pp. 547–562. Cited by: §I, §II.
 [30] (2013) The finite element method: theory, implementation, and applications. Springer Publishing Company, Incorporated. External Links: ISBN 3642332862, 9783642332869 Cited by: §I, §II.
 [31] Folding deformable objects using predictive simulation and trajectory optimization. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6000–6006. Cited by: §I.
 [32] (2007) Real time simulation of nonlinear tissue response in virtual surgery using the point collocationbased method of finite spheres. Computer Methods in Applied Mechanics and Engineering 196 (3132), pp. 3011–3024. Cited by: §II.
 [33] (201311) Fast simulation of massspring systems. ACM Transactions on Graphics 32 (6), pp. 209:1–7. Note: Proceedings of ACM SIGGRAPH Asia 2013, Hong Kong External Links: Link Cited by: §IV.
 [34] (2008) Visualizing data using tsne. Journal of machine learning research 9 (Nov), pp. 2579–2605. Cited by: §VIC.
 [35] (201707) Convolutional neural networks on surfaces via seamless toric covers. ACM Trans. Graph. 36 (4), pp. 71:1–71:10. External Links: ISSN 07300301, Link, Document Cited by: §II.
 [36] (2015Sept) VoxNet: a 3d convolutional neural network for realtime object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vol. , pp. 922–928. External Links: Document, ISSN Cited by: §II.
 [37] (2017) Muscularhydrostat computers: physical reservoir computing for octopusinspired soft robots. In Brain Evolution by Design, pp. 403–414. Cited by: §II.
 [38] (201211) Adaptive anisotropic remeshing for cloth simulation. ACM Transactions on Graphics 31 (6), pp. 147:1–10. Note: Proceedings of ACM SIGGRAPH Asia 2012, Singapore Cited by: §I, 2nd item, §IV, Fig. 4, §VC, §VIB, §VIF, TABLE I, §VI.
 [39] (201311) Sparse localized deformation components. ACM Trans. Graph. 32 (6), pp. 179:1–179:10. External Links: ISSN 07300301 Cited by: §I, §II, §VID, TABLE II.
 [40] (2018) Hierarchical cloth simulation using deep neural networks. arXiv:1802.03168 . Cited by: §II.
 [41] (201805) Realtime planning for highdof deformable bodies using twostage learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), Vol. , pp. 1–8. Cited by: §II, §II.
 [42] (2016) Robot motion planning for pouring liquids. In TwentySixth International Conference on Automated Planning and Scheduling, Cited by: §I.
 [43] (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434 . Cited by: §II.
 [44] (2018) Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), Cited by: §I.
 [45] (2007) Asrigidaspossible surface modeling. In Proceedings of the Fifth Eurographics Symposium on Geometry Processing, SGP ’07, AirelaVille, Switzerland, Switzerland, pp. 109–116. External Links: ISBN 9783905673463, Link Cited by: §VA.
 [46] (2018) Meshbased autoencoders for localized deformation component analysis. In Proceedings of the ThirtySecond AAAI Conference on Artificial Intelligence, Cited by: §I, §II, §VIB, §VID, TABLE I, TABLE II, TABLE III.
 [47] (201102) A perception correlated comparison method for dynamic meshes. IEEE Transactions on Visualization and Computer Graphics 17 (2), pp. 220–230. External Links: ISSN 10772626, Link, Document Cited by: §VIB, §VII.
 [48] (2015) Deformation capture and modeling of soft objects. ACM Transactions on Graphics(Proc. of SIGGRAPH 2015) 34 (4), pp. 94:1–94:12. Cited by: §II.
 [49] (200802) Gaussian process dynamical models for human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 30 (2), pp. 283–298. External Links: Document, ISSN 01628828 Cited by: §II.
 [50] (2018) Latentspace physics: towards learning the temporal evolution of fluid flow. arXiv:1802.10123 . Cited by: §II, §VB.
 [51] (2018) TempoGAN: a temporally coherent, volumetric gan for superresolution fluid flow. ACM Transactions on Graphics (TOG) 37 (4), pp. 95. Cited by: §II.
 [52] (2016) Perspective transformer nets: learning singleview 3d object reconstruction without 3d supervision. arXiv:1612.00814 . Cited by: §II.
 [53] (2014) Optimization equivalence of divergences improves neighbor embedding. In International Conference on Machine Learning, pp. 460–468. Cited by: §VIC.
 [54] (200906) Nonrigid shape recovery by gaussian process regression. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, Vol. , pp. 1319–1326. External Links: Document, ISSN 10636919 Cited by: §I, §II.
 [55] (2004) Sparse principal component analysis. J. Comp. Graph. Statistics 15, pp. 2006. Cited by: §I, §II.