_{︎
Tools Used}

_{Visual Studio Code
Google Workspace
Figma
Adobe  Photoshop
Adobe  Lightroom
Adobe Premiere Pro}

︎
Project Overview

Issue addressed


Despite efforts like 3D movies, widespread solutions are rare. As augmented and mixed reality gain traction in our daily lives, the need for innovative depth-screening methods grows. 

Solution

Layered transparent LCD screens that enables a deeper screen experience.

How & Process

01.Prototype
02.User Test

We are developing a novel prototype that tackles these challenges by utilizing layered LCD screens arranged in configuration with each other. In this project, we showcase our prototype and delve into the visual elements that produce this effect, drawing from user testing results and a paper set to be published in April 2024.

︎
ABSTRACT

Representing depth on a surface has been a design concern since at least the Renaissance.

And yet, despite this long history, 3D displays have not become widely adopted. However,with the emergence of mixed reality applications, the development of new modalities forproviding depth to displays has become even more relevant. We present a novel design that addresses these challenges by employing layered transparent LCD screens in configuration, which we call Deep Surface Liquid Crystal Displays (DS-LCD). DS-LCDs require no additional artifacts such as anaglyph glasses. Drawing on both stereoscopic and volumetric approaches, we present the DS-LCD prototype and discuss the visual elements that create the “deep surface” effect. We then describe the initial software applications that were developed to validate the prototype, as well as our initial user studies. We conclude with speculation about future work and applications.


Keywords: Extended Reality, Augmented Reality, Mixed Reality, 3D Display, Stereoscopic Display,Volumetric Display, Liquid Crystal Display

The ability to create the illusion of depth upon a two-dimensional surface has long been a concern of artists and technologists interested in the verisimilitude of visual representation. Painting was arguably the first attempt in the modern era to describe what we might now call virtual or augmented reality. Painters were and are world makers. By the early 15th century, artists were developing new strategies for representing deep, volumetric space on flat surfaces. These works of art are now esteemed because one finds, in the best instances, a masterful integration of linear perspective, atmospheric perspective, relative size, and chiaroscuro that create a space within which a story can take place.

In the 20th century, with the development of film and television, moving images could be orchestrated into time-based experiences that added greater realism than static two-dimensional representations such as painting.  As these technologies matured, storytellers and technologists continued to explore ways of creating more immersive experiences.

Since then, the most common strategy for creating 3D effects has been the use of stereoscopic images, which produce a 3D effect by taking advantage of binocular vision.[1] This is usually achieved by splitting the image into opposing colors, typically red and cyan, and filtering each color through anaglyph glasses. When the images are reassembled in the viewer’s perception, it gives the illusion of the image popping off the surface.  However, while anaglyph glasses have become widespread, public sentiment has not been overwhelmingly positive.  A 2023 survey shows that demand for traditional 3D technologies has tapered off; however, it projects that with the integration of improved, anaglyph glasses free technologies, global demand, particularly driven by Asian markets, is projected to nearly quadruple for 3D cinema. [2] Many virtual reality headsets use variations of this technique, and therefore require the use of wearables, typically mounted onto the head and in front of the eyes.

Autostereoscopy, which does not require the use of anaglyph glasses or other similar wearables, employs a similar technique. Glasses are unnecessary because lenticular lenses or parallax barriers on the surface separate the image for each eye to process based on relative position. Binocular vision produces the effect relative to the position of the viewer. Because these kinds of screens are dependent on specific viewing zones and the relative position of the viewer, outside of which the effect is not seen, autostereoscopy also has significant limitations.[3]

Volumetric displays and other holographic effects, such as Pepper’s Ghost, are yet another approach to creating 3D user experiences.[4][5] Typically, some kind of substrate is  needed to project upon, which need not be flat, but does need to maintain a calibrated position between the viewer and the projected image.  Like stereoscopic images, volumetric displays also have limitations in terms of viewing angles and resolution.

Deep Surface Liquid Crystal Displays (DS-LCD), as we call the prototypes presented here, offer an alternative design strategy with several advantages. DS-LCDs can be used as a kind of volumetric display in the z axis, employing the shallow space produced in the gap between the two displays. The two (or more) layers work as substrates upon which slices of the image can be produced. But DS-LCDs can also work stereoscopically, in the y and x axes, by taking advantage of binocular effects. It doesn’t  require anaglyph glasses, and so it can produce a deep surface experience that is integrated directly into the built environment. This strategy eschews the pitfalls of wearables in general, such as cumbersome AR and VR devices that attach to the head, and returns to the paradigm that painters developed centuries ago.

Figure 1: Prototype of the DS-LCD

Prototype - Hardware

Deep Surface Liquid Crystal Displays work much like a traditional display. However, instead of a single LCD display, our prototype features two LCD displays, one layered on top of the other with a small gap in between. The rear LCD screen is a standard unmodified LCD display. The front LCD display is modified by removing the opaque backing layer, allowing for the liquid crystal layer and polarizing layers to remain transparent. (Figure 1 and Figure 2) This allows for the layering, or doubling, of additional visual content onto the background screen. These LCD displays are mounted onto an adjustable aluminium frame, which allowed us to vary and test the size of the gap between screens. Initial testing determined that a gap of between 1.75 inches and 2.75 inches between the screens was most optimal for realizing the 3D effect. (Figure 3)

Figure 2: DS-LCD prototype viewed from front.

Figure 3: DS-LCD prototype viewed from side and rear.

In order to understand how media might be displayed onto the layered displays, and best enhance the depth of field effect, we developed software that could: 1) Play two videos files (e.g. mp4) on the two displays simultaneously. 2) Separate the background field from the foreground image using OpenCV, and play both videos simultaneously on both displays.  3) Play two video files in sequence in order to create a “leaping” effect of a smaller background image moving forward toward the viewer and becoming a larger foreground image on the transparent display.


The first Python program - videoplayer - simply played two video files at the same time on the two displays. This was accomplished by creating two windows for each video, defining the x and y coordinates for each window, and resizing and aligning each frame of the video to conform to the correct frame dimensions. This first iteration of the prototype allowed for the further development and enhancement of the deep surface effect.


The second Python program – videoextractor – took in a file path to read to and a file path to write to, then went through each frame of the input file and removed the background using the OpenCV module cv2. The extracted foreground image could then be played on the front transparent display, while the background image could then be played on the rear display.  In videoconferencing applications, this separates the image of a person from the background, giving a deeper spatial effect. (Figure 4)

Figure 4: Two displays, one transparent and one opaque, are layered in configuration to produce greater depth (depth not to scale).

The third Python program – videoleaper – split a video into two files that could be played on both screens sequentially. A smaller image on the rear screen jumps forward onto the front screen, increasing slightly in size to further enhance the effect of an object coming towards the viewer. We initially tested this effect by creating a simulation of a bouncing ball that “leaped” between screens at five second intervals, growing larger on the front display and smaller on the rear display. From this initial test, we developed and tested other media to further enhance the effect. (Figure 5 and Figure 6)

Figure 5: The plane leaps from the rear display to the front display.

Figure 6: The ball leaps from the rear display to the front display and back to the rear display in a circular motion

Using these programs, a series of demos were developed, including:

1. Video conferencing demo: Using vidoextrator, a local user engages with a remote user in a typical video conferencing scenario. The video image of the local user was separated into a foreground image and a background image, each displayed on its respective display.
2. Airplane animation demo: Using a single video file and videoleaper, a plane flying forward jumps from the rear display to the front display, slightly enlarging the image, and moving from display to display.
3. Ball animation demo: A bouncing ball jumps to the front screen as it enlarges and back to the back screen as it shrinks again using the same technique as the airplane animation demo.

︎
USABILITY TESTING

Ten graduate and undergraduate students from the Rhode Island School of Design (RISD) and Brown University participated in the user tests and surveys. They possessed a basic understanding of 2D digital screens and an interest in digital experiences. None of the participants had a disability that might affect the user tests. The user tests were conducted twice with different groups, and survey forms were distributed to all participants.

Methodology

The user testing was conducted in a dark room to ensure optimal performance of the DS-LCD prototype, with participants seated in chairs positioned in front of the DS-LCD. Two of the user tests were conducted under the same conditions. Participants were given a general introduction to the DS-LCD project, explaining that the display system is made of two layers of 2D digital screens, enabling depth beyond that of a flat screen and providing an enhanced dimensional experience. Two scenarios were presented to the user groups.

01. The first scenario presented subjects with a video conferencing experience.  We asked participants to have a 2 to 3 minute video conference conversation, with one participant using a remote laptop in an adjacent room, and one participant using the prototype. Subjects were asked to compare the video conferencing experience of a traditional display with the experience of using a DS-LCD. Users followed up by filling out a survey. Users were asked if a more volumetric presentation of remote users enhanced their telepresence experience, and generally made for more natural interactions. Users were also asked if the ability to move content between the displays, from front to back and back to front, allowed for greater facility with presenting content on the screen.

02. The second scenario explored the experience of the 3D effects when animated between the displays. Using the videoleaper asset, participants were shown a series of animated videos where objects came towards the viewer and moved away from the user, shifting in size and between the rear display and the front display. For example, we showed users a ball animation that moved in a circular motion horizontally to the user, moving from the rear screen to the front screen. We also demonstrated an animation of a plane that flies toward the subject, moving from the rear screen to the front screen.  

01. Videoconferencing Scenario

Four participants (40%) reported being highly engaged (4 out of 5 on the scale) with this scenario, while two participants (20%) reported being even more highly engaged (5 out of 5). Additionally, all ten participants (100%) agreed that it was easier to empathize with the speaker on the DS-LCD teleconferences compared to traditional teleconferences. Their ratings varied, with 60% giving it a score of 3 out of 5, 30% rating it 4 out of 5, and 10% giving it the highest rating of 5 out of 5.

Furthermore, all ten participants (100%) expressed a preference for meeting via DS-LCD videoconferencing than with traditional approaches. Their ratings varied, with 50% giving it a score of 3 out of 5, 20% rating it 4 out of 5, and 30% giving it the highest rating of 5 out of 5.

Overall, when comparing the videoconferencing experience with DS-LCDs against traditional teleconferencing, participants indicated that they were more engaged, and enjoyed the meetings more than traditional modalities.

02. Animation Scenario

Two scenes are overlapped at the same time, enabling a dynamic entertainment experience for the visual art media.

The animations playing and background are each different visual animations. They could be overlapped altogether, which could provide a visual experience never experienced before.

Eight participants (80%) reported feeling engaged while watching visuals with the DS-LCD, rating their engagement as 4 out of 5 on the scale, and one participant (10%) rated it as 5 out of 5. This finding is particularly significant considering that only 30% of participants reported feeling engaged with traditional modalities (the sum of ratings 4 and 5 out of 5), whereas 90% of participants reported feeling engaged with the DS-LCD experience (the sum of ratings 4 and 5 out of 5). This indicates a notable preference for a higher level of engagement with the DS-LCD visuals compared to the control.

These initial prototypes validated the idea that layered transparent LCD displays can produce greater depth by combining both stereoscopic and volumetric approaches. The effect is something akin to looking into clear water from the shore.  On the surface plane of the  x and y axes, unadulterated images or  stereoscopic images can be used to enhance binocular effects, while the shallow space in the z axis - in the physical gap between displays - can produce a very convincing volumetric effect. This iteration of the DS-LCD prototype challenges us to consider what other ways we might further integrate these two approaches in order to enhance the spatial effect?

In future iterations, one can imagine introducing more transparent LCD layers in the gap between the rear and front displays as a way to increase the resolution of the volumetric output. By further altering the images, including stereoscopic manipulation, one can image further enhancing the effect.

Mixed reality is also an area of interest for future interactions. Beyond the videoconferencing scenarios already explored, one can imagine a role for DS-LCDs in telepresence applications where the boundary between remote and local space is less distinct. [6] At different scales and within different contexts, one can imagine a role for DS-LCDs that further blurs the boundaries between remote and local spaces, and is fully integrated into the built environment.

︎
ACKNOWLEDGMENT

The authors would like to acknowledge the Brown University Computer Science Department for supporting this project.

︎

REFERENCES

[1] Kooi, F., & Toet, A. (2004). Visual comfort of binocular and 3D displays. Displays, 25(2–3), 99-108. https://doi.org/10.1016/j.displa.2004.07.004

[2]YouGov. (2011). YouGov survey results. https://cdn.yougov.com/today_uk_import/yg-archives-yougov-3dfilms-230911.pdf

[3] Mphepo, W. (2020). Stereoscopy and autostereoscopy. In Mixed Reality and Three-Dimensional Computer Graphics (Chapter 9). DOI: 10.5772/intechopen.92633

[4] England, J. (2018). Science of Pepper's Ghost Illusion. Cosmos Magazine. Retrieved from https://cosmosmagazine.com/science/physics/the-science-behind-the-peppers-ghost-illusion/

[5] Gonsher, I., Feng, P., McArn, T., & Christenson, A. (2023, July). Designing for wide adoption: An inexpensive and accessible extended reality (XR) device. In International Conference on Human-Computer Interaction (pp. 185-191). Cham: Springer Nature Switzerland.

[6] Gonsher, I. (2024). The telepresence of furniture in extended reality. Interactions, 31(2), 44-48.