Yevsey Gutman1*, Nadejda Krasnorutskaya2 and Gary Gutman3
1GEI Systems, Minneapolis, USA
2ITMO, Saint Petersburg, Russia
3Columbus State College, Columbus, USA
*Corresponding Author: Yevsey Gutman, GEI Systems, Minneapolis, USA.
Received: February 18, 2021; Published: : March 17, 2021
Citation: Yevsey Gutman., et al. “Challenging Issues in Development and Integration of Modern Kinematics with Components and Images Obtained from Different Sources in Biomechanical Applications”. Acta Scientific Dental Sciences 5.4 (2021): 102-113.
In spite of the existing and successful development of CAD/CAM methods for mechanical applications, the ability to model the motion of biomechanical systems still presents many challenges. The reason for those challenges is in complexity of the nature of components to be used for the projects, specifically based on the origin of the parts: where are they coming from. Depending on the biomechanical application it could be as many as three different sources for the parts to be included in to the “assembled” system. They are: (1) “standard” parts which can be designed using the conventional CAD system, (2) parts which can be obtained only through the process of surface scanning; and (3) components which are the product of CT scanning with volumetric property. An Advanced method is discussed here which can deal in case of biomechanical systems to address the weaknesses of the Standard Approach in use for purely mechanical systems.
Keywords: CAD/CAM; CT Scanning; CAD System; Modern Kinematics; Biomechanical Applications
To analyze the kinematics of a mechanical system [9], as represented in a virtual environment, the researcher has to consider:
The structure of this paper can be summarized as follows:
We now compare how the standard and advanced approaches accomplish these tasks.
Subject selectionThe study included 30 systemically healthy volunteers those who fulfilled the following inclusion criteria.
The environment for virtual development in the standard approach is a commercial CAD package, such as SolidWorks, Alibre Design, Autodesk, PTC. These packages have efficient methods of creation most of “standard” components, such as those which can be virtually fabricated by CAD programs based on existing sketches and drawings with dimensions and other properties.
By the well-established process of “mating”, these packages allow the user to assemble such components. Then, both individual and assembled components are rendered on a screen by proprietary software as a part of the same package As the user drags one of “independent” components across the screen to a new position, the motion of the system with the rest of parts is calculated and established for the dependent components by a “solver”.
The advantage of these standard packages is that they require minimal expertise on the part of the researcher! Let us move to their disadvantages in the case of biomechanical systems.
Disadvantages of the standard approachThe components of a biomechanical are unlike the components of a typical mechanical system, such as an automobile engine or machine tool building products: They are difficult to specify analytically due to the complex nature of the origin.
In a biomechanical system, the components come from the following three sources:
The first group of parts can be handled by any standard CAD packages; the other two require richer data structures.
Performance degradation of a standard solver for biomechanical systemsEven if the standard packages have a work-around to include these more complex classes of components, the native solver built in the standard CAD/CAM packages, experiences serious degradation in performance, as evidenced by a decrease in calculation and rendering speed.
Lack of effective user interface, parameter control and monitoringIn the standard CAD/CAM packages [12], it is at best awkward to associate a parameter to the relative position of a component (outside of usage of mouse). This makes it difficult to numerically monitor and control the state of a complex system and even more difficult to pass this state to any other process for analysis. Except in the case of systems with one degree of freedom, where the state of the system can be changed by simply dragging a component, it is awkward to make the system perform to specified parameters. In particular, the user lacks an effective interface that can control the system, except in the simplest of cases. Thus, it is difficult to efficiently test the system in realistic situations.
Advantages of the advanced approachThe advanced approach remedies the weakness of the standard approach by:
The advanced approach utilizes a sophisticated interface [11,13,14] between the end user and the model on the screen, with screen handling in the advanced approach done by a home-made module to render images. This is successfully achieved for models where all information about the kinematics of the entire mechanism is known. For each assembly a displacement model must be developed, requiring intimate knowledge of the kinematics of the model and experience developing displacement models for multi-DOF systems. A comparison of the two approaches from the point of view of the end-user should include their experience in:
So, as it was mentioned above, there are three major classes of components in biomechanical systems:
In the advanced approach, models for all components are represented in the common STL format (Figure 1) regardless of their origin.
Figure 1: Flow diagram for parts developed by different methods.
Having represented the components in a common format, we can consider the environment in which those models should be assembled for virtual articulation.
Based on our experience and requirements for the performance to be achieved during simulation, we conclude that the best environment is the DirectX package by Microsoft [11].
This package has not just an efficient way to deal with rendering components on the screen, but also a full library of function calls to solve many graphics issues. It allows the developer to run a collision detection task between two meshes in real time without any performance degradation. And it is well “connected” to the hardware of the system in form of video card such as NVIDIA through the Windows OS and graphic drivers.
The only “side effect” of using DirectX software is in the requirement to present those components/meshes obtained for each model in DirectX format known as dot x (.x). That requires one more step to convert all model files so far presented in STL format to .x format (Figure 2). That conversion has been done using a very efficient PolyTrans program from Okino Corporation [10].
Figure 2: Conversion from STL to DirectX format.
We now illustrate these steps by examples of components generated in different environments.
First, we are going to start from the part or better to say a group of parts or sub-assembly which can be developed using a CAD system where each component has been created by a standard CAD package. In general, DirectX has the capability of developing those parts within its own system but this process is more efficient in standard CAD packages environment. That is why, all “standard” (conventional parts) are created by CAD systems. In our example, since it demonstrates an assembly of a fully adjustable articulator, the following parts: base, some moving parts to mount the upper and lower models, facebow and other “standard” components are made by Solid Works (Figure 3) and then saved in STL format. Some of those parts are discussed in figure.
Figure 3: Sample of parts developed by standard CAD.
To display components on the screen, a front panel with all man-machine controls and dialogs has been developed (Figure 4). This panel is based on microsoft visual studio IDE and visual basic.
Figure 4: Fully adjustable articulator displayed on the front panel (no models displayed).
System requirements dictate that a DirectX SDK is installed prior to any development. By pushing a “Run” button from the main menu, we put the program in to an indefinite “play” loop which can be terminated by selecting a “Stop” option. Within this loop we have to display all parts and components on the screen in proper positions and perform the rendering process.
Sub PlayGame ()
Do While GameActive
If JoyStickModeSwitch Then JoyCallBack
MyDX8Graph.view Vwx, Vwy, Vwz 'View point: Vx, Vy, Vz
MyDX8Graph.Camera Cwx, Cwy, Cwz ‘Camera point: Cx, Cy, Cz
MyDX8Graph.changeLight LPosition 'twx, twy, twz ': Tx, Ty, Tz
MyDX8Graph.Render
DoEvents
Loop
EndGame
Unload Me
End Sub
For best results the system uses a multi-buffer technique where one buffer is used to display the results and the other is to prepare the screen with the (part of DirectX setup process).
The final critical issue is to present the motion of the parts as an assembled mechanism with mating conditions which reflect its kinematics: to develop a displacement model of the system.
Providing an advanced solver specifically adapted to biomechanical systemsThe advanced approach fine tunes the kinematics of the model of a biomechanical system to the needs of the components. This is done by adapting the displacement method which uses extended rotation and translation matrices to develop the evolution of the six DOF of a mechanical system in three-dimensional space to create the solver engine. To do this, the Advanced Approach must establish local and world coordinates of components of the biomechanical system which are manipulated by the displacement method using the specific kinematics of the system.
World coordinate system in DirectX: In DirectX, a part to be displayed on the screen goes through a rendering process using a matrix to transform world coordinates. This matrix is applied to all parts. It has six parameters: x, y, z, α, β, γ. Using those six parameters we display a part in different positions and orientations on the screen relative to the world coordinate system.
We consider those six parameters as six DOF for each part which is modeled in DirectX space. If those DOFs are set to zero, the world matrix is the identity matrix, and the part is displayed with its local (part) coordinate system on the top of world coordinate: The part looks as it would be depicted in its original (local) coordinate system.
For any change in DOF parameters, the matrix of world coordinate transformation established for each individual part is recalculated and this transformation is applied to the part for the proper placement on the screen. To use the world matrix efficiently it is important to properly establish the local coordinate system during creation of each part. Let us consider it now.
Local coordinates for solid parts: It is relatively easy to establish a local coordinate system for the model developed using a CAD package as for a solid part. This local coordinate system is preserved in the process of saving it in STL format and then later in a .X mesh file. If the first look at the model in DirectX environment is not what you have planned for, you can go back to the CAD program and insert a local reference coordinate system where it is more practical, repeating the process of saving this model in STL and then .X.
Local coordinates for unconventional parts: This where the Advanced Approach shows its advantage! The system considered for articulation in this project contains other parts (Figure 5) which cannot be produced (or produced efficiently) by CAD system but can be established by scanning of them (both jaws models) on a surface scanning device.
Figure 5: Upper and lower models in mesh form.
Those models (Figure 6) are obtained from the scanner in form of Point Clouds. In our case here they are the models of a lower and upper jaws. The process of obtaining them is described above: a) scanning the models, b) collecting the Point Cloud for each part, c) converting it into an STL mesh, d) converting STL into the DirectX mesh with extension X.
Figure 6: Scanners developed by laser design corporation and parts for scanning.
So, with those two models included into the rendering engine the front control panel would look like (Figure 7). For those parts generated by point cloud the information about the local coordinate system (the origin of the part) is as critical as for the standard parts.
Figure 7: Articulator shown with point cloud parts (upper and lower jaws).
Parts generated from point clouds: The task of establishing the local coordinate system for parts obtained from a point cloud is more complicated due to the fact that we do not know the coordinate system of the scanner. We circumvent this as follows.
The entire Cloud is recorded in this coordinate system. To establish (identify) the local coordinate system we have to use a calibration fixture with known geometry and dimensions which would be mounted on the “table” of the scanner and collect the point cloud for this model.
Calibration process for point cloud parts: The goal of this task is to establish/recover the local coordinate system associated with the part during the scanning procedure. The local coordinate system during the scanning is technically unknown and depends on the design and configuration of the scanning device. If we take the part “as is” with its original (but unknown) local coordinal system we then display the point cloud (the mesh represented this cloud) under the world system within DirectX rendering loop. And we might be surprised to find the part not exactly where you would expect to see it (Figure 8). That is happening due to the “wrong” orientation of the model’s coordinate system.
Figure 8: Specimen on lower arm.
To fix this problem, we have to use a special fixture which can be mounted on the “table” of the scanner. The fixture can be attached to the scanner through a magnetic base for easy and quick removal. This fixture (let us call it FIX for short) has to be scanned and then, its point cloud be converted to a FIX.STL and finally to FIX.X file. The virtual display of this fixture on the same arm in DirectX will look like on figure 9.
Figure 9: Before calibration.
As it can be seen and explained from the figure 8 and 9, the model for FIX.X does not have the right coordinate system. But since we know all dimensional parameters of the model with high precision we can “recover” the right coordinate system for the FIX model so it can be used for the rest models developed from the Point Cloud on this scanner.
To achieve that, first, we have to create a “probe” function using a set of standard DirectX utility toolbox. This function allows us to probe (digitize) any surface (mesh) within DirectX environment by placing the cursor on/over the top of the surface (Figure 10) which we would like to get coordinates of point from. Secondly, we have to develop a program called “fixture calibration” to establish the missing coordinate system. The control panel developed for this procedure is shown on figure 11.
Figure 10: Process to calibrate point cloud coordinate system.
Figure 11: Control panel used to conduct the calibration.
The algorithm for this procedure is based on establishing the position and orientation of three planes which are the three sides of the model FIX. To define them we select three points on each face (side of the model) by probing them with a virtual mouse. The first plane X has three points designated as Point1 or P1 for short, P2 and P3 (Figure 10 and 12).
Figure 12: Points on virtual model during calibration.
By pressing the button “Get Plane” we establish the position of Plane X. Repeating this step for Plane Y and Plane Z (three points on each face: P4, P5 and P6 for plane Y and P7, P8 and P9 for plane Z) and pushing the button “Get Plane” after each face we will establish all three planes and as a result the corner point P of the fixture model. This corner point P is the intersection of all three planes (Figure 13). The next two steps will determine the origin of this local coordinate system (press button “Get Origin”) and then “Cal” to complete the calibration procedure (Figure 14).
Figure 13: Calibration panel after the first plane X is established.
Figure 14: Final view of control panel before re-calibration of the system.
By applying the calibration matrix to the original model (press “Recalculate Origin Matrix and Apply”) we correct the position of the model in DirectX environment.
So, the matrix we have established during this procedure will be used to convert every image of the mesh obtained from the point cloud on this scanning device. In our case, after replacing the fixture with the real part and the right (corrected) local coordinate system we have on figure 15 and 16.
Figure 15: Fixture with the right coord system.
Figure 16: Part on the arm after calibration.
These improvements provide the advanced approach with an implementation of the displacement method for biomechanical systems which effectively give is a superior solver engine which render it superior to the standard approach for such systems.
Parts generated from CT scanner: Those parts are represented in DICOM format and by selecting a desired value of density we can create a “part” (Figure 17) which can be saved in STL format.
Figure 17: Images obtained during CT scan in DICOM environment.
The same issue, as for parts obtained from Point Cloud, we have to face here about establishing the local coordinate system, but the imaging software (Figure 18) comes with utility which can do it interactively right on the model in DICOM environment.
Figure 18: CT Image with utility tools (from Anatomage package).
Volumetric scanning: Manipulating the images of the Virtual models of bio-mechanical samples on the screen, we are dealing with the digital representation of those models, more precisely, with its surface images or mesh images. That means actually that we don’t know much about anything inside the model or under the “skin”. To look “deeper” a CT scans technology known to covers many different test/procedures such as scanning parts of the body can be used. It is based on new Cone Beam technology and called CBCT. Usually, it is a relatively small device designed with particular interest for dentistry, very fast (takes about 20 secs. per scan) and it is less harmful for the environment and patient (1/17 of a regular CT dosage of X-ray radiation).
To extend the surface scanning process to volumetric models obtained from CBCT scans the following steps are required:
In steps 2 and 3, the DICOM information is separated into two images and converted into the STL format using digital imaging software such as “Create Model”.
Fusing STL and DICOM images: During this process, the two images obtained from the CBCT scans and converted to the STL format should be fused/mated with the corresponding STL image of the models obtained during a surface scanning of casts. Using a set of simple fixtures and "similar" features shared between the corresponding parts, it can be considered as a superimposing process by which the mesh of the lower cast is laid over the image of the mandible (Figure 19) obtained from the CT scan as well as the upper jaw laid over the maxilla area of the cranial image.
Figure 19: Articulation of two fused models: STL (from point Cloud) and DICOM.
Effective user interfaces: The advanced approach provides flexible user interfaces through the DirectX environment: One of the features which makes use of DirectX very efficient and productive for special applications is in its interface with the model in practically real time. This very flexible interaction between the user and the model on the screen can be achieved with ability to use a tool box with components such as scroll bars, digital meters, graphs and many other controls with are the part of Microsoft IDE. That includes Visual Basic and Visual C++.
The other feature of DirectX which separates this approach from traditional modeling of the kinematics is the ability to use USB peripheral devices like joystick for interaction with the model.