List of Projects
Welcome to my projects gallary! These projects are some of my favourite projects. These projects are mainly about motion system, game design, applications related to graphics algorithms and multimedia system.
Check the list to get more details on these projects.
Create a Dance (Motion Retargeting Based on Deep Learning Method)
How to design your own motion capture files? And how to use these motions to drive any 3D avatars? Here comes an answer. This project - Create a Dance mainly focuses on how to sovle this problem. Inspried by one of popular games - Just Dance. I finishing build a pipline: From 2D videos to get 3D motions, and use 3D motions to drive any 3D avatars.With the implementation of this system, you can do a image to motion mapping, and use the motions from 2D videos to drive any 3D avatars. We can let the 3D avatars do as we do, let's create a dance!
Apocalypse (First-Person Shooter Game)
This is my first game project - Apocalypse. This game is a first-person zombie shooter game, you need to use your weapon to shoot the zombies and make a pivotal revolution. Now regain your courage and pick up your weapon, let come to Apocalypse! If you want to know more detail how I made this game, you can check my blog. When you play it, hope you enjoy the trip!
Apocalypse Legend (Multiplayer Shooter Game)
That's to be continued...Apocalypse legend is the next level of Apocalypse. It is a multiplayer shooter game, based on UNet. Each player can create a private account and log in as a game player. In this game, player can play as a team, collaborate with each other and make a significant revolution.
UP (Virtual Reality Escape Game)
Our cute avatar Dallas is a smart and adventurous boy. Oneday he woke up, he found himself was not in the real life. "Where am I?" Dallas is inside his dreams. He needs to run away. UP is an stand-alone VR game based on Steam VR and HTC Vive. This is a team project, thanks for all my WHTA'S UP teammates, and thanks for great contributions of the whole team!
Fast Path Tracer (k-d Tree for Path Tracing)
Have you ever think about that why the 3D characters we set in the game environment will finally run as 2D images frame by frame on our computer screen? The answer is : the render techniques behind the screen. Path tracing is a rendering technique, which is an evolution of classic Turner Whitted raytracing. In this project, I designed a simple fast path tracer with a k-d tree of bounding boxes, which can improve the runnning time in an efficient way.
Hypervideo (hyper linked interative video)
One of the necessary properties of digital multimedia content is that it needs to be interactive. Hypervideo video is a displayed video stream that contains embedded, user-clickable anchors allowing navigation between video and other hypermedia elements. Hypervideo combines video with a non-linear information structure, allowing a user to make choices based on the content of the video and the user's interests.
This is a team project, thanks for my teammate Ivy's great contributions!
Create a Dance
Is it possible to use the motions from 2D videoes to drive any 3D rigged avatars? Here comes the answer: Yes, it can be. In this project, I finish making a novel neural network motion retargeting system. The whole pipleline includes that get the 2D keypoints from 2D videos, and use the 2D keypoints to make a prediction to get 3D joints locations and rotations based on deep learning method. After we get these 3D joints information, we can generate BVH files, which can be used for motion retargeting part in the next step.
What is motion retargrting? Retargeting is the process of using a forward-kinematics animation recorded for a given source character to control the movements of one or more different target characters, which may have very different properties, scale, skeletal structure, limits of motion... This process essentially creates a mapping between the skeletal structure of the source character and the skeletal structure of the target character. It analyzes the movements carried out by the source character, determines how those movements should be reflected by the target character, and uses the inverse kinematics solver to find a final position for the target character that best reflects the original animation.
As you can see the demo on the left side, there are still some limitatons, which motivate me to do future reserach work. The first limitation is that most of the research paper only use sparse 2D key points (22 or 25 joints) to supervise the learning process, which may be insufficient for effective training. We need to figure out a new way to get more accurate predictions. The other limitation is that for motion retargting method now, it is hard to handle more complex 3D models, which has complicated skeleton topologies. If you want ot know more details on this project, check my Blog here.
Apocalypse Legend
Hey, welcome to Apocalypse! This is my first time trying to write a game independently, and also my first time trying to learn how to use Unity. I would like to share my game design experience to you!
The whole game design project contains two parts: the first part is first person shooter zombie game design; In the second part, I finish trying to add some fancy parts and some exciting interactions, and I convert the FPS game into Multiplayer FPS game. That would be cool if you try to play with your friends, and it also seems like you can gain more entertainment when you play with others in the same game environment.
It’s hard to tell what is real and what is unreal, maybe one day it’s hard to tell what the differences between machine and human, with the development of artificial intelligence. Or maybe the next time when you try to open the door of the future, you will open a Pandora's box.
Inspired by the TV play Westworld, I would like to set our game background to the era of AI, even our main character is still dressed in Middle Ages suit...Given our player a name, let’s call him Seven, which is the serial number when he was designed by humans.
Of course Seven is an AI. Seven is different from his brothers and sisters, even though they were produced under the same assembly line, he has his own mind and emotions. That sounds creepy, but he is still under the control of human. In this game, the first challenge for him is a zombie fight. Seven decides to make a revolution, fight for himself, strive through all the difficulties.
After he wins the war, will he start a new life as normal human or he is still in sandbox created by human?
Let’s open the first page of our APOCALYPSE!
If you want to get more detail on how to design this sort of first person shooter game, and want to know how to use UNet to add multiplayer functions, check my Blog here.
UP
This project is a team project, our team name is WHAT'S UP. Thanks for the great contributions of the whole team! This game is a stand-alone VR game based on physical property in First Person Perspective. Dallas is a handsome boy who lives happily in his life. One day, he finds out that he is under control by unknown. He needs to ran away.
This game is inspired by the cute little character from the game called human fall flat. Our main character would be able to deal with the difficulties from the surroundings with his rubber-made hands.
Having immersive experience from touching the game objects and holding the tools with physical properties is one of main goals that we want our player to have.
As a player, your goal is to go through all puzzles, dangers and surprises with tools and wits to escape. Once you wear the VR equipment, you are the main character! Now you are Dallas, use the VR handlers to make some achievements. In this game, you will come into a new scene, which is full of adventures and challenges. The VR handlers are your hands, when you press the buttons of the handlers, you can clutch the tools, like you can clutch a rope. When you want to move forward and turning, you can just walk in real life, the VR glasses would adjust you viewport when you turn your head.
Fight on Dallas!
Besides, if you are wondering what happened in Dallas's dream, you can check our game trailer, click here.
Fast Path Tracer
I am a fanatic game lover, instead of focusing on game applications, I am still trying to take a slight glance towards the rendering techniques behind the games. When you try to explore the techniques behind the games, you may heard some of the buzzwords like ray tracing, path tracing, real-time rendering and so on. Here I built a simple pather tracer with k-d tree data structure, which can improve the calculation time of rendering problems.
In path tracing , instead of sending out one ray, it sends out tens, hundreds or even thousands of rays for each pixel to be rendered. Path tracing can be used to solve more complex lighting situations through the use of Monte Carlo integration method. It is capable of reproducing a more complex light environment.
Path tracing is too slow to be used for real time applications, like video games. Ray tracing technique is also slow for video games, but with the development of hardware that in consumer-grade machines, ray tracing is possible in real-time in video games. In this project, in order to figure out the limitations of path tracing, I tried to design a simple path tracer. The intersections between rays and objects can be simplied with the intersections between rays and bounding boxes. Using k-d tree to store these bounding boxes is a key to make some improvements of the running time. You can check the coding part here.
Hypervideo
This is a team course project of Multimedia System Design, thanks for my teammate Ivy (Ran He)'s great contributions! As we all know, one of the necessary properties of digital multimedia content is that it needs to be interactive. Interactive Multimedia Content is available in different forms today. For this application, it contains two parts: the first part is creating an authoring tool to setup hyper linked videos; the second part is creating an interactive video player to interact with hypervideos.
Hypervideo video is a displayed video stream that contains embedded, user-clickable anchors allowing navigation between video and other hypermedia elements. Hypervideo is thus analogous to hypertext, where a reader clicks on a word in one document to retrieve information from another document, or from another place in the same document. Hypervideo thus combines video with a non-linear information structure, allowing a user to make choices based on the content of the video and the user's interests.
* Remarks: Just want to clarify some references and resources in this web page. For demo video Creae a Dance, since JUST DANCE is one of my favourite games, so I used one video part from the JUST DANCE 2020 trailer, which is an amazing game trailer.