Special to S&T Review

Display, interactions and applications of immersive metaverse: Progress and outlooks

  • HE Zehao ,
  • CAO Liangcai
Expand
  • State Key Laboratory of Precision Measurement Technology and Instrument, Department of Precision Instruments, Tsinghua University, Beijing 100084, China

Received date: 2022-12-05

  Revised date: 2023-02-10

  Online published: 2023-03-27

Abstract

Some critical aspects of metaverse are briefed. The significance of immersion to metaverse is highlighted. Efforts to improve the sense of immersion are investigated. The principles of some immersive display technologies including head-mounted display (HMD), head-up display (HUD) and large size display are introduced and the corresponding prospects are envisioned. The importance of immersive interaction on metaverse is analyzed. Values of computer graphics, artificial intelligence and machine vision for metaverse are induced. The potential applications of metaverse are predicted based on the mentioned features.

Cite this article

HE Zehao , CAO Liangcai . Display, interactions and applications of immersive metaverse: Progress and outlooks[J]. Science & Technology Review, 2023 , 41(5) : 6 -14 . DOI: 10.3981/j.issn.1000-7857.2023.05.001

References

[1] Stephenson N. Snow crash[M]. New York: Bantam Spectra, 1992.
[2] Mystakidis S. Metaverse[J]. Encyclopedia, 2022, 2(1): 486-497.
[3] Zhao Y H, Jiang J J, Chen Y, et al. Metaverse: Perspectives from graphics, interactions and visualization[J]. Visual Informatics, 2022, 6(1): 56-67.
[4] Park S M, Kim Y G. A metaverse: Taxonomy, components, applications, and open challenges[J]. IEEE Access,2022, 10: 4209-4251.
[5] 王文喜, 周芳, 万月亮, 等 . 元宇宙技术综述[J]. 工程科学学报, 2022, 44(4): 744-756.
[6] Fan Y C, Lee C M, Lee M Y, et al. 3D display and interactive technology in metaverse[J]. ITE Transactions on Media Technology and Applications, 2022, 10(4): 179-189.
[7] Kanematsu H, Barry D M, Shirai T, et al. Virtual experiences of metaverse using mobile type head-mounted displays and their applicability[J]. Procedia Computer Science, 2022, 207: 4428-4433.
[8] 曹良才, 何泽浩, 刘珂瑄, 等. 元宇宙中的动态全息三维显示: 发展与挑战[J]. 红外与激光工程, 2022, 51(1):267-281.
[9] 张汉乐, 邢妍, 胡晓帅, 等 . 面向元宇宙的集成成像3D显示技术进展[J]. 指挥与控制学报, 2022, 8(3): 239-248.
[10] Lawrence J, Goldman D, Achar S, et al. Project starline:A high-fidelity telepresence system[J]. ACM Transactions on Graphics, 2021, 40(6): 242.
[11] Cheng D W, Wang Q W, Liu Y, et al. Design and manu⁃facture AR head-mounted displays: A review and outlook[J]. Light: Advanced Manufacturing, 2021, 2(3):350-369.
[12] Itoh Y, Langlotz T, Sutton J, et al. Towards indistinguishable augmented reality: A survey on optical see-through head-mounted displays[J]. ACM Computing Surveys,2022, 54(6): 120.
[13] Tian X, Liu R, Wang Z Y, et al. High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision[J]. Information Fusion, 2022, 77:19-28.
[14] Watanabe H, Omura T, Okaichi N, et al. Full-parallax three-dimensional display based on light field reproduction[J]. Optical Review, 2022, 29(4): 366-374.
[15] Wang Y D, Sang X Z, Xing S J, et al. Three-dimensional light-field display with enhanced horizontal viewing angle by introducing a new lenticular lens array[J]. Optics Communications, 2020, 477: 126327.
[16] Ravishankar J, Sharma M, Gopalakrishnan P. A flexible coding scheme based on block Krylov subspace approximation for light field displays with stacked multiplicative layers[J]. Sensors, 2021, 21(13): 4574.
[17] Smalley D E, Nygaard E, Squire K, et al. A photophoretic-trap volumetric display[J]. Nature, 2018, 553: 486-490.
[18] Hua J Y, Yi D H, Qiao W, et al. Multiview holographic 3D display based on blazed Fresnel DOE[J]. Optics Communications, 2020, 472: 125829.
[19] Blanche P A. Holography, and the future of 3D display[J]. Light: Advanced Manufacturing, 2021, 2(4): 446-459.
[20] He Z Z, Li K X, Sui X M, et al. Holographic 3D display using depth maps generated by 2D-to-3D rendering approach[J]. Applied Sciences, 2021, 11(21): 9889.
[21] Mu C T, Lin W T, Chen C H. Zoomable head-up display with the integration of holographic and geometrical imaging[J]. Optics Express, 2020, 28(24): 35716-35723.
[22] Skirnewskaja J, Wilkinson T D. Automotive holographic head-up displays[J]. Advanced Materials, 2022, 34(19):2110463.
[23] Lv Z, Liu J, Yang Y. Dual-view and multi-content head-up display using a single picture generation unit and two-layer volume holographic grating[J]. IEEE Photonics Journal, 2022, 14(4): 7035208.
[24] Valtakari N V, Hooge I T C, Viktorsson C, et al. Eye tracking in human interaction: Possibilities and limitations[J]. Behavior Research Methods, 2021, 53(4): 1592-1608.
[25] Drakopoulos P, Koulieris G A, Mania K. Eye tracking interaction on unmodified mobile VR headsets using the selfie camera[J]. ACM Transactions on Applied Perception, 2021, 18(3): 11.
[26] Pai Y S, Dingler T, Kunze K. Assessing hands-free interactions for VR using eye gaze and electromyography [J]. Virtual Reality, 2019, 23(2): 119-131.
[27] 朱富丽, 杨磊, 申玉斌, 等 . 基于眼动特征的视觉交互状态分类方法研究[J]. 航天医学与医学工程, 2021, 34(6): 426-431.
[28] Pavlovic V I, Sharma R, Huang T S. Visual interpretation of hand gestures for human-computer interaction: A review[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1997, 19(7): 677-695.
[29] Li Y, Huang J, Tian F, et al. Gesture interaction in virtual reality[J]. Virtual Reality & Intelligent Hardware,2019, 1(1): 84-112.
[30] Cheok M J, Omar Z, Jaward M H. A review of hand gesture and sign language recognition techniques[J]. International Journal of Machine Learning and Cybernetics,2019, 10(1): 131-153.
[31] 林星雨, 邢妍, 张汉乐, 等 . 基于 Leap Motion 手势识别的悬浮真3D显示实时交互系统[J]. 液晶与显示, 2022,37(5): 654-659.
[32] Yen H Y. Smart wearable devices as a psychological intervention for healthy lifestyle and quality of life: A randomized controlled trial[J]. Quality of Life Research,2021, 30(3): 791-802.
[33] Yen H Y, Liao Y, Huang H Y. Smart wearable device users' behavior is essential for physical activity improvement[J]. International Journal of Behavioral Medicine,2022, 29(3): 278-285.
[34] Ringeval M, Wagner G, Denford J, et al. Fitbit-based interventions for healthy lifestyle outcomes: Systematic review and meta-analysis[J]. Journal of medical Internet research, 2020, 22(10): e23954.
[35] 刘任高 . 具有复杂人机交互功能的可穿戴产品设计研究[D]. 南京: 南京航空航天大学, 2021.
[36] Liu J N, Qian H, Xiao A P, et al. Human-machine interaction based on voice[J]. AASRI Procedia, 2012, 3: 583-588.
[37] Wellmann F, Caumon G. 3-D structural geological models: Concepts, methods, and uncertainties[J]. Advances in Geophysics, 2018, 59: 1-121.
[38] Paulsen J, Ali T M L, Collas P. Computational 3D genome modeling using Chrom3D[J]. Nature, 2018, 13:1137-1152.
[39] Cao Xuan, Geng Z, Li T. Dictionary-based light field acquisition using sparse camera array[J]. Optics Express,2014, 22(20): 24081-24095.
[40] Serabyn E. Pupil segmentation in the light-field camera and its relation to 3D object positions and the reconstructed depth of field[J]. Applied Optics, 2019, 58(5):A273-A282.
[41] He L, Liu K, He Z, et al. Three-dimensional holographic communication system for the metaverse[J]. Optics Communications, 2023, 526: 128894.
[42] Yu P F, Yang S R, Chen S Y. Accuracy improvement of time-of-flight depth measurement by combination of a high-resolution color camera[J]. Applied Optics, 2020,59(35): 11104-11111.
[43] Coffey V C. Machine Vision: The eyes of industry 4.0[J].Optics & Photonics News, 2018, 29(7): 42-49.
[44] Ren Z, Fang F, Li Z, et al. Intelligent evaluation for lens optical performance based on machine vision[J]. Optics Express, 2022, 30(15): 26251-26265.
[45] 颉永鹏 . 基于机器视觉的工业机器人目标识别和定位研究[D]. 沈阳: 沈阳工业大学, 2022.
[46] Bregler C. Motion capture technology for entertainment[J]. IEEE Signal Processing Magazine, 2007, 24(6): 158-160.
[47] Chan J C P, Leung H, Tang J K T, et al. A virtual reality dance training system using motion capture technology[J]. IEEE Transactions on Learning Technologies, 2011,4(2): 187-195.
[48] Choe N, Zhao H Y, Qiu S, et al. A sensor-to-segment calibration method for motion capture system based on low cost MIMU[J]. Measurement, 2019, 131: 490-500.
[49] Luck M, Aylett R. Applying artificial intelligence to virtual reality: Intelligent virtual environments[J]. Applied Artificial Intelligence, 2010, 14(1): 3-32.
[50] Qin F W, Li L Y, Gao S M, et al. A deep learning approach to the classification of 3D CAD models[J]. Journal of Zhejiang University Science C, 2014, 15(2): 91-106.
[51] Zhu Z T, Wang X G, Bai S, et al. Deep learning representation using autoencoder for 3D shape retrieval[J].Neurocomputing, 2016, 204: 41-50.
[52] Cheng F C, Zhang H, Fan W J, et al. Image recognition technology based on deep learning[J]. Wireless Personal Communications, 2018, 102(2): 1917-1933.
[53] Luciano L, Hamza A B. A global geometric framework for 3D shape retrieval using deep learning[J]. Computers & Graphics, 2019, 79: 14-23.
[54] 武强, 季雪庭, 吕琳媛. 元宇宙中的人工智能技术与应用[J]. 智能科学与技术学报, 2022, 4(3): 324-334.
[55] 何泽浩, 隋晓萌, 赵燕, 等 . 基于全息光学的虚拟现实与增强现实技术进展[J]. 科技导报, 2018, 36(9): 8-17.
Outlines

/