Augmented Reality
Please note, HoloLens AR is currently only supported on Windows Desktop when using the C++ API and the DirectX 11 driver.
For detailed instructions on setting up AR/VR with Visualize, please see the Getting Started section.
Visualize currently supports AR in both HoloLens native and HoloLens remoting applications. A HoloLens native app is built for the HoloLens architecture and deployed to the device. A HoloLens remoting application uses a remotely connected computer to render the image and sends it over a WiFi network to the HoloLens. While there are many similarities in how these applications are structured, we will address them separately.
HoloLens Native
In order to add native HoloLens support to your application, the following items are required:
A HoloLens headset
HoloLens is currently only supported on Windows Desktop, when using the C++ API, and the DirectX 11 driver.
For a reference implementation of HoloLens native in a Visualize application, you can refer to the HPS Holographic Sandbox, bundled with your copy of Visualize.
1. Set Up PC and HoloLens
You must enable Developer mode on both the PC and HoloLens, and pair the devices, to allow deployment from Visual Studio on the PC to the HoloLens device. For details, see Microsoft’s documentation here: https://docs.microsoft.com/en-us/windows/mixed-reality/using-visual-studio
2. Initializing Application in Visualize
The fundamentally essential steps are to overload the Windows::ApplicationModel::Core::IFrameworkView and Windows::ApplicationModel::Core::IFrameworkViewSource classes, and call CoreApplication::Run(yourIFrameworkViewSource).
The IFrameworkViewSource class only needs to create a new IFrameworkView object. See the AppViewSource class in AppView.h and AppView.cpp for an example.
The IFrameworkView class should overload the various triggers that can come from the HoloLens: ViewActivated, Suspending, Resuming, WindowClosed, etc. See the AppView class in AppView.h and AppView.cpp for an example.
We recommend separating the HPS related activity into a separate class - in the sample, we use HolographicSandboxMain.
3. Initializing HoloLens in Visualize
In the IFrameworkView::Initialize method, you can set up HPS to handle the HoloLens device. This includes an InitPicture event handler to maintain the HoloLens context:
Sample code
HolographicSandboxMain::HolographicSandboxMain()
{
// Step 1. Set up an offscreen window for HoloLens
HPS::OffScreenWindowOptionsKit oswok;
oswok.SetDriver(HPS::Window::Driver::DirectX11);
oswok.SetHardwareResident(true);
oswok.SetFramebufferRetention(false);
oswok.SetAntiAliasCapable(false);
oswok.SetOpacity(0.0f);
m_window_key = HPS::Database::CreateOffScreenWindow(1268, 720, oswok);
m_window_aspect = 1268.0f / 720;
// Step 2. Set HPS driver bit to render to HoloLens
HPS_Driver_Set_HoloLens_Bit(m_window_key, 0);
// Step 3. Create an Init Picture Handler (see below)
m_window_key.SetDriverEventHandler(m_init_picture_handler, HPS::Object::ClassID<HPS::InitPictureEvent>());
m_window_key.UpdateWithNotifier().Wait();
assert(m_init_picture_handler.dx11_device.Get());
// Step 4 create HPS canvas, view, and model
auto portfolio = HPS::Database::CreatePortfolio();
m_canvas = HPS::Factory::CreateCanvas(m_window_key, portfolio);
m_view = HPS::Factory::CreateView();
m_canvas.AttachViewAsLayout(m_view);
m_view.GetSegmentKey().GetDrawingAttributeControl().SetWorldHandedness(HPS::Drawing::Handedness::Right);
m_view.GetSegmentKey().GetMaterialMappingControl().SetWindowColor(HPS::RGBAColor(0.0f, 0.0f, 0.0f, 0.0f));
m_view.GetSegmentKey().GetCullingControl().SetFrustum(true).SetExtent(0);
auto model = HPS::Factory::CreateModel();
m_view.AttachModel(model);
// Your Initialization code here
}
// InitPictureHandler class is an HPS class with just one method - Handle
class InitPictureHandler : public HPS::DriverEventHandler
{
public:
Microsoft::WRL::ComPtr<ID3D11Device> dx11_device;
ID3D11Texture2D * target = nullptr;
HPS::MatrixKit proj_left;
HPS::MatrixKit proj_right;
HPS::MatrixKit view_left;
HPS::MatrixKit view_right;
void Handle(HPS::DriverEvent const * in_event);
};
// This method populates the projection and view matrices for each eye
void InitPictureHandler::Handle(HPS::DriverEvent const * in_event)
{
auto e = (HPS::InitPictureEvent const *)in_event;
static bool first_update = true;
if (first_update)
{
auto ctx = (ID3D11DeviceContext*)e->GetGraphicsContext();
ctx->GetDevice(&dx11_device);
first_update = false;
}
else
{
e->SetRenderSurface((HPS::OpaqueHandle)target);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ProjectionLeft, proj_left);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ProjectionRight, proj_right);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ViewLeft, view_left);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ViewRight, view_right);
}
}
4. Start a Render Loop
In the IFrameworkView::Run method, we process events coming in from the HoloLens, update the scene, and render to a new frame. We do this by requesting a new frame from the HoloLens, rendering into that frame, and then presenting the frame back to the device:
Sample code
// m_main is of class HolographicSandboxMain
void AppView::Run()
{
while (!m_windowClosed)
{
if (m_windowVisible && (m_holographicSpace != nullptr))
{
CoreWindow::GetForCurrentThread()->Dispatcher->ProcessEvents(CoreProcessEventsOption::ProcessAllIfPresent);
HolographicFrame^ holographicFrame = m_main->Update();
if (m_main->Render(holographicFrame))
{
// The holographic frame has an API that presents the swap chain for each
HolographicFramePresentResult present_result = holographicFrame->PresentUsingCurrentPrediction(HolographicFramePresentWaitBehavior::DoNotWaitForFrameToFinish);
}
}
else
{
CoreWindow::GetForCurrentThread()->Dispatcher->ProcessEvents(CoreProcessEventsOption::ProcessOneAndAllPending);
}
}
}
// Updates the application state once per frame.
HolographicFrame^ HolographicSandboxMain::Update()
{
// The HolographicFrame has information that the app needs in order
// to update and render the current frame. The app begins each new
// frame by calling CreateNextFrame.
HolographicFrame^ holographicFrame = m_holographicSpace->CreateNextFrame();
// Get a prediction of where holographic cameras will be when this frame
// is presented.
HolographicFramePrediction^ prediction = holographicFrame->CurrentPrediction;
// Next, we get a coordinate system from the attached frame of reference that is
// associated with the current frame. Later, this coordinate system is used for
// for creating the stereo view matrices when rendering the sample content.
SpatialCoordinateSystem^ currentCoordinateSystem = m_referenceFrame->CoordinateSystem;
// We complete the frame update by using information about our content positioning
// to set the focus point.
for (auto cameraPose : prediction->CameraPoses)
{
// The HolographicCameraRenderingParameters class provides access to set
// the image stabilization parameters.
HolographicCameraRenderingParameters^ renderingParameters = holographicFrame->GetRenderingParameters(cameraPose);
// SetFocusPoint informs the system about a specific point in your scene to
// prioritize for image stabilization. The focus point is set independently
// for each holographic camera.
// You should set the focus point near the content that the user is looking at.
// In this example, we put the focus point at the center of the sample hologram,
// since that is the only hologram available for the user to focus on.
// You can also set the relative velocity and facing of that content; the sample
// hologram is at a fixed point so we only need to indicate its position.
renderingParameters->SetFocusPoint(currentCoordinateSystem, m_focus_point);
}
// The holographic frame will be used to get up-to-date view and projection matrices and
// to present the swap chain.
return holographicFrame;
}
// Renders the current frame to each holographic camera, according to the
// current application and spatial positioning state. Returns true if the
// frame was rendered to at least one camera.
bool HolographicSandboxMain::Render(Windows::Graphics::Holographic::HolographicFrame^ holographicFrame)
{
// Up-to-date frame predictions enhance the effectiveness of image stabilization and
// allow more accurate positioning of holograms.
holographicFrame->UpdateCurrentPrediction();
auto prediction = holographicFrame->CurrentPrediction;
bool atLeastOneCameraRendered = false;
for (auto cameraPose : prediction->CameraPoses)
{
// The view and projection matrices for each holographic camera will change
// every frame.
auto & cameraProjectionTransform = cameraPose->ProjectionTransform;
// If TryGetViewTransform returns a null pointer, that means the pose and coordinate
// system cannot be understood relative to one another; content cannot be rendered
// in this coordinate system for the duration of the current frame.
// This usually means that positional tracking is not active for the current frame, in
// which case it is possible to use a SpatialLocatorAttachedFrameOfReference to render
// content that is not world-locked instead.
Platform::IBox<HolographicStereoTransform>^ viewTransformContainer = cameraPose->TryGetViewTransform(m_referenceFrame->CoordinateSystem);
if (viewTransformContainer)
{
auto & viewCoordinateSystemTransform = viewTransformContainer->Value;
UpdateCameraParams(
(float const *)&cameraProjectionTransform.Left,
(float const *)&cameraProjectionTransform.Right,
(float const *)&viewCoordinateSystemTransform.Left,
(float const *)&viewCoordinateSystemTransform.Right,
HPS::Window::Driver::DirectX11,
HPS::Drawing::Handedness::Right,
m_window_aspect,
m_init_picture_handler.proj_left,
m_init_picture_handler.proj_right,
m_init_picture_handler.view_left,
m_init_picture_handler.view_right,
m_camera_kit);
}
HolographicCameraRenderingParameters^ camera_parameters = holographicFrame->GetRenderingParameters(cameraPose);
ComPtr<ID3D11Resource> resource;
ThrowIfFailed(GetDXGIInterfaceFromObject(camera_parameters->Direct3D11BackBuffer, IID_PPV_ARGS(&resource)));
ComPtr<ID3D11Texture2D> camera_backbuffer;
ThrowIfFailed(resource.As(&camera_backbuffer));
m_init_picture_handler.target = camera_backbuffer.Get();
m_window_key.UpdateWithNotifier().Wait();
m_init_picture_handler.target = nullptr;
// docs say to use last frame's camera for culling
m_view.GetSegmentKey().SetCamera(m_camera_kit);
atLeastOneCameraRendered = true;
}
return atLeastOneCameraRendered;
}
HoloLens Remoting
In order to add remote HoloLens support to your application, the following items are required:
A HoloLens headset
HoloLens is currently only supported on Windows desktop, when using the C++ API, and the DirectX 11 driver.
For a reference implementation of HoloLens remote in a Visualize application, you can refer to the HPS Holographic Remoting Sandbox, bundled with your copy of Visualize.
1. Set Up PC and HoloLens
Connect the PC and HoloLens to the same network. Open the Holographic Remoting application on the HoloLens. This should display an IP address, which you will need to provide to your code. The Remoting sample app uses a command line argument.
2. Initializing Application in Visualize
The fundamental steps to HoloLens remoting are similar to HoloLens native development. You must create an HPS offscreen window, connect to the device, and then start your update/render loop. See the main function in main.cpp for details. The difference is that instead of overloading functions in the IFrameworkView class, you must instead initialize a HolographicStreamerHelpers object, and assign ConnectedEvents to its members: OnConnected, OnDisconnected, etc.
Sample code
#include <d3d11_1.h>
#include <dxgi1_3.h>
#include <assert.h>
#include <HolographicStreamerHelpers.h>
#include <windows.graphics.directx.direct3d11.interop.h>
#pragma comment(lib, "d3d11.lib")
#include "hps.h"
#include "sprk.h"
#include "hoops_license.h"
#include "../vr_shared/arvr.h"
using namespace Microsoft::Holographic;
using namespace Microsoft::WRL;
using namespace Microsoft::WRL::Wrappers;
using namespace Windows::Graphics::Holographic;
using namespace Windows::Graphics::DirectX::Direct3D11;
using namespace Windows::Perception::Spatial;
static const int VIDEO_FRAME_WIDTH = 1280;
static const int VIDEO_FRAME_HEIGHT = 720;
static const float window_aspect = (float)VIDEO_FRAME_WIDTH / VIDEO_FRAME_HEIGHT;
static HPS::Canvas canvas;
static HPS::CameraKit camera_kit;
static HPS::OffScreenWindowKey window_key;
static HPS::View view;
static HPS::SegmentKey help_seg;
static Platform::String^ ip_address = L"127.0.0.1";
static Windows::Graphics::Holographic::HolographicSpace^ holographic_space;
static HolographicStreamerHelpers^ streamer_helpers;
static Windows::Perception::Spatial::SpatialStationaryFrameOfReference^ reference_frame;
static bool connected = false;
//InitPictureHandler class is an HPS class with just one method - Handle
//This method populates the projection and view matrices for each eye
class InitPictureHandler : public HPS::DriverEventHandler
{
public:
ComPtr<ID3D11Device> dx11_device;
ID3D11Texture2D * target = nullptr;
HPS::MatrixKit proj_left;
HPS::MatrixKit proj_right;
HPS::MatrixKit view_left;
HPS::MatrixKit view_right;
void Handle(HPS::DriverEvent const * in_event)
{
auto e = (HPS::InitPictureEvent const *)in_event;
static bool first_update = true;
if (first_update)
{
auto ctx = (ID3D11DeviceContext*)e->GetGraphicsContext();
ctx->GetDevice(&dx11_device);
first_update = false;
}
else
{
e->SetRenderSurface((HPS::OpaqueHandle)target);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ProjectionLeft, proj_left);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ProjectionRight, proj_right);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ViewLeft, view_left);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ViewRight, view_right);
}
}
};
static void Connect()
{
if (connected)
{
streamer_helpers->Disconnect();
connected = false;
}
else
{
try
{
streamer_helpers->Connect(ip_address->Data(), 8001);
}
catch (Platform::Exception^ ex)
{
DebugLog(L"Connect failed with hr = 0x%08X", ex->HResult);
}
}
}
int main(Platform::Array<Platform::String^>^ args)
{
RoInitializeWrapper roinit(RO_INIT_MULTITHREADED);
ThrowIfFailed(HRESULT(roinit));
Platform::String^ filename;
float scale = 1.0f;
for (uint32_t i = 1; i < args->Length; ++i)
{
if (args[i] == "-ip" && i + 1 < args->Length)
ip_address = args[++i]; //Get the ip address of the device
}
auto world = new HPS::World(HOOPS_LICENSE);
HWND hwnd = CreateWin32Window(VIDEO_FRAME_WIDTH, VIDEO_FRAME_HEIGHT, L"HPS Holographic Remoting Sandbox", false, wndproc);
HPS::OffScreenWindowOptionsKit oswok;
oswok.SetDriver(HPS::Window::Driver::DirectX11);
oswok.SetHardwareResident(true);
oswok.SetFramebufferRetention(false);
oswok.SetAntiAliasCapable(true);
oswok.SetOpacity(0.0f);
window_key = HPS::Database::CreateOffScreenWindow(VIDEO_FRAME_WIDTH, VIDEO_FRAME_HEIGHT, oswok);
HPS_Driver_Set_HoloLens_Bit(window_key, (HPS::WindowHandle)hwnd);
auto portfolio = HPS::Database::CreatePortfolio();
canvas = HPS::Factory::CreateCanvas(window_key, portfolio);
InitPictureHandler init_picture_handler;
window_key.SetDriverEventHandler(init_picture_handler, HPS::Object::ClassID<HPS::InitPictureEvent>());
window_key.UpdateWithNotifier().Wait();
assert(init_picture_handler.dx11_device.Get());
view = HPS::Factory::CreateView();
view.GetSegmentKey().GetDrawingAttributeControl().SetWorldHandedness(HPS::Drawing::Handedness::Right);
view.GetSegmentKey().GetMaterialMappingControl().SetWindowColor(HPS::RGBAColor(0.0f, 0.0f, 0.0f, 0.0f));
view.GetSegmentKey().GetCullingControl().SetFrustum(true).SetExtent(0);
//This segment will be displayed on the PC
help_seg = view.GetSegmentKey().Subsegment("help_text");
help_seg.InsertText(HPS::Point(0, 0, 0), "Press space bar to connect");
help_seg.GetTextAttributeControl().SetAlignment(HPS::Text::Alignment::Center).SetSize(0.05f, HPS::Text::SizeUnits::SubscreenRelative);
help_seg.GetVisibilityControl().SetText(true);
help_seg.GetMaterialMappingControl().SetTextColor(HPS::RGBAColor::White());
canvas.AttachViewAsLayout(view);
auto model = HPS::Factory::CreateModel();
view.AttachModel(model);
Windows::Foundation::Numerics::float3 focus_point;
model.GetSegmentKey().GetPerformanceControl().
SetStaticModel(HPS::Performance::StaticModel::Attribute).
SetDisplayLists(HPS::Performance::DisplayLists::Segment);
streamer_helpers = ref new HolographicStreamerHelpers();
streamer_helpers->CreateStreamer(init_picture_handler.dx11_device.Get());
streamer_helpers->SetVideoFrameSize(VIDEO_FRAME_WIDTH, VIDEO_FRAME_HEIGHT);
streamer_helpers->SetMaxBitrate(4 * 1024);
streamer_helpers->SetEnableAudio(false);
holographic_space = streamer_helpers->HolographicSpace;
{
// Acquire the DXGI interface for the Direct3D device.
ComPtr<IDXGIDevice3> dxgi_device;
ThrowIfFailed(init_picture_handler.dx11_device.As<IDXGIDevice3>(&dxgi_device));
// Wrap the native device using a WinRT interop object.
auto interop_device = CreateDirect3DDevice(dxgi_device.Get());
holographic_space->SetDirect3D11Device(interop_device);
}
reference_frame = SpatialLocator::GetDefault()->CreateStationaryFrameOfReferenceAtCurrentLocation();
streamer_helpers->OnConnected += ref new ConnectedEvent(
[]()
{
DebugLog(L"Connected");
connected = true;
help_seg.GetVisibilityControl().SetEverything(false);
});
//Handle disconnection
Platform::WeakReference streamerHelpersWeakRef = Platform::WeakReference(streamer_helpers);
streamer_helpers->OnDisconnected += ref new DisconnectedEvent(
[streamerHelpersWeakRef](_In_ HolographicStreamerConnectionFailureReason failureReason)
{
DebugLog(L"Disconnected with reason %d", failureReason);
connected = false;
// Reconnect if this is a transient failure.
if (failureReason == HolographicStreamerConnectionFailureReason::Unreachable ||
failureReason == HolographicStreamerConnectionFailureReason::ConnectionLost)
{
DebugLog(L"Reconnecting...");
try
{
auto helpersResolved = streamerHelpersWeakRef.Resolve<HolographicStreamerHelpers>();
if (helpersResolved)
helpersResolved->Connect(ip_address->Data(), 8001);
else
DebugLog(L"Failed to reconnect because a disconnect has already occurred.\n");
}
catch (Platform::Exception^ ex)
{
DebugLog(L"Connect failed with hr = 0x%08X", ex->HResult);
}
}
else
{
DebugLog(L"Disconnected with unrecoverable error, not attempting to reconnect.");
}
});
streamer_helpers->OnSendFrame += ref new SendFrameEvent(
[](_In_ const ComPtr<ID3D11Texture2D>& spTexture, _In_ FrameMetadata metadata)
{
});
// Main Update/Render loop
// See hololens_remote_2.cpp for an example
.....
}
3. Connect to HoloLens From PC
The HolographicStreamerHelpers class contains most of the ways of interacting with the HoloLens. In particular, it has a Connect function that you must call in order to start receiving poses and sending frames to the device. You pass this the IP address of the HoloLens headset, and a port number - by default, 8001. We can also get the HolographicSpace from the HolographicStreamerHelpers object, which will be how our render loop receives poses from the device.
Sample code
#include <d3d11_1.h>
#include <dxgi1_3.h>
#include <assert.h>
#include <HolographicStreamerHelpers.h>
#include <windows.graphics.directx.direct3d11.interop.h>
#pragma comment(lib, "d3d11.lib")
#include "hps.h"
#include "sprk.h"
#include "hoops_license.h"
#include "../vr_shared/arvr.h"
using namespace Microsoft::Holographic;
using namespace Microsoft::WRL;
using namespace Microsoft::WRL::Wrappers;
using namespace Windows::Graphics::Holographic;
using namespace Windows::Graphics::DirectX::Direct3D11;
using namespace Windows::Perception::Spatial;
static const int VIDEO_FRAME_WIDTH = 1280;
static const int VIDEO_FRAME_HEIGHT = 720;
static const float window_aspect = (float)VIDEO_FRAME_WIDTH / VIDEO_FRAME_HEIGHT;
static HPS::Canvas canvas;
static HPS::CameraKit camera_kit;
static HPS::OffScreenWindowKey window_key;
static HPS::View view;
static HPS::SegmentKey help_seg;
static Platform::String^ ip_address = L"127.0.0.1";
static Windows::Graphics::Holographic::HolographicSpace^ holographic_space;
static HolographicStreamerHelpers^ streamer_helpers;
static Windows::Perception::Spatial::SpatialStationaryFrameOfReference^ reference_frame;
static bool connected = false;
//InitPictureHandler class is an HPS class with just one method - Handle
//This method populates the projection and view matrices for each eye
class InitPictureHandler : public HPS::DriverEventHandler
{
public:
ComPtr<ID3D11Device> dx11_device;
ID3D11Texture2D * target = nullptr;
HPS::MatrixKit proj_left;
HPS::MatrixKit proj_right;
HPS::MatrixKit view_left;
HPS::MatrixKit view_right;
void Handle(HPS::DriverEvent const * in_event)
{
auto e = (HPS::InitPictureEvent const *)in_event;
static bool first_update = true;
if (first_update)
{
auto ctx = (ID3D11DeviceContext*)e->GetGraphicsContext();
ctx->GetDevice(&dx11_device);
first_update = false;
}
else
{
e->SetRenderSurface((HPS::OpaqueHandle)target);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ProjectionLeft, proj_left);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ProjectionRight, proj_right);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ViewLeft, view_left);
e->SetStereoMatrix(HPS::DriverEvent::StereoMatrix::ViewRight, view_right);
}
}
};
static void Connect()
{
if (connected)
{
streamer_helpers->Disconnect();
connected = false;
}
else
{
try
{
streamer_helpers->Connect(ip_address->Data(), 8001);
}
catch (Platform::Exception^ ex)
{
DebugLog(L"Connect failed with hr = 0x%08X", ex->HResult);
}
}
}
int main(Platform::Array<Platform::String^>^ args)
{
RoInitializeWrapper roinit(RO_INIT_MULTITHREADED);
ThrowIfFailed(HRESULT(roinit));
Platform::String^ filename;
float scale = 1.0f;
for (uint32_t i = 1; i < args->Length; ++i)
{
if (args[i] == "-ip" && i + 1 < args->Length)
ip_address = args[++i]; //Get the ip address of the device
}
auto world = new HPS::World(HOOPS_LICENSE);
HWND hwnd = CreateWin32Window(VIDEO_FRAME_WIDTH, VIDEO_FRAME_HEIGHT, L"HPS Holographic Remoting Sandbox", false, wndproc);
HPS::OffScreenWindowOptionsKit oswok;
oswok.SetDriver(HPS::Window::Driver::DirectX11);
oswok.SetHardwareResident(true);
oswok.SetFramebufferRetention(false);
oswok.SetAntiAliasCapable(true);
oswok.SetOpacity(0.0f);
window_key = HPS::Database::CreateOffScreenWindow(VIDEO_FRAME_WIDTH, VIDEO_FRAME_HEIGHT, oswok);
HPS_Driver_Set_HoloLens_Bit(window_key, (HPS::WindowHandle)hwnd);
auto portfolio = HPS::Database::CreatePortfolio();
canvas = HPS::Factory::CreateCanvas(window_key, portfolio);
InitPictureHandler init_picture_handler;
window_key.SetDriverEventHandler(init_picture_handler, HPS::Object::ClassID<HPS::InitPictureEvent>());
window_key.UpdateWithNotifier().Wait();
assert(init_picture_handler.dx11_device.Get());
view = HPS::Factory::CreateView();
view.GetSegmentKey().GetDrawingAttributeControl().SetWorldHandedness(HPS::Drawing::Handedness::Right);
view.GetSegmentKey().GetMaterialMappingControl().SetWindowColor(HPS::RGBAColor(0.0f, 0.0f, 0.0f, 0.0f));
view.GetSegmentKey().GetCullingControl().SetFrustum(true).SetExtent(0);
//This segment will be displayed on the PC
help_seg = view.GetSegmentKey().Subsegment("help_text");
help_seg.InsertText(HPS::Point(0, 0, 0), "Press space bar to connect");
help_seg.GetTextAttributeControl().SetAlignment(HPS::Text::Alignment::Center).SetSize(0.05f, HPS::Text::SizeUnits::SubscreenRelative);
help_seg.GetVisibilityControl().SetText(true);
help_seg.GetMaterialMappingControl().SetTextColor(HPS::RGBAColor::White());
canvas.AttachViewAsLayout(view);
auto model = HPS::Factory::CreateModel();
view.AttachModel(model);
Windows::Foundation::Numerics::float3 focus_point;
model.GetSegmentKey().GetPerformanceControl().
SetStaticModel(HPS::Performance::StaticModel::Attribute).
SetDisplayLists(HPS::Performance::DisplayLists::Segment);
streamer_helpers = ref new HolographicStreamerHelpers();
streamer_helpers->CreateStreamer(init_picture_handler.dx11_device.Get());
streamer_helpers->SetVideoFrameSize(VIDEO_FRAME_WIDTH, VIDEO_FRAME_HEIGHT);
streamer_helpers->SetMaxBitrate(4 * 1024);
streamer_helpers->SetEnableAudio(false);
holographic_space = streamer_helpers->HolographicSpace;
{
// Acquire the DXGI interface for the Direct3D device.
ComPtr<IDXGIDevice3> dxgi_device;
ThrowIfFailed(init_picture_handler.dx11_device.As<IDXGIDevice3>(&dxgi_device));
// Wrap the native device using a WinRT interop object.
auto interop_device = CreateDirect3DDevice(dxgi_device.Get());
holographic_space->SetDirect3D11Device(interop_device);
}
reference_frame = SpatialLocator::GetDefault()->CreateStationaryFrameOfReferenceAtCurrentLocation();
streamer_helpers->OnConnected += ref new ConnectedEvent(
[]()
{
DebugLog(L"Connected");
connected = true;
help_seg.GetVisibilityControl().SetEverything(false);
});
//Handle disconnection
Platform::WeakReference streamerHelpersWeakRef = Platform::WeakReference(streamer_helpers);
streamer_helpers->OnDisconnected += ref new DisconnectedEvent(
[streamerHelpersWeakRef](_In_ HolographicStreamerConnectionFailureReason failureReason)
{
DebugLog(L"Disconnected with reason %d", failureReason);
connected = false;
// Reconnect if this is a transient failure.
if (failureReason == HolographicStreamerConnectionFailureReason::Unreachable ||
failureReason == HolographicStreamerConnectionFailureReason::ConnectionLost)
{
DebugLog(L"Reconnecting...");
try
{
auto helpersResolved = streamerHelpersWeakRef.Resolve<HolographicStreamerHelpers>();
if (helpersResolved)
helpersResolved->Connect(ip_address->Data(), 8001);
else
DebugLog(L"Failed to reconnect because a disconnect has already occurred.\n");
}
catch (Platform::Exception^ ex)
{
DebugLog(L"Connect failed with hr = 0x%08X", ex->HResult);
}
}
else
{
DebugLog(L"Disconnected with unrecoverable error, not attempting to reconnect.");
}
});
streamer_helpers->OnSendFrame += ref new SendFrameEvent(
[](_In_ const ComPtr<ID3D11Texture2D>& spTexture, _In_ FrameMetadata metadata)
{
});
// Main Update/Render loop
// See hololens_remote_2.cpp for an example
.....
}
4. Render Loop
We then start the main loop. We request a prediction of where the cameras will be from the device, then set those cameras in HPS. We then update the HPS window, and Present the result to the HoloLens. See the main loop for an example. In particular, see the Update and Render functions.
Sample code
#include <d3d11_1.h>
#include <dxgi1_3.h>
#include <assert.h>
#include <HolographicStreamerHelpers.h>
#include <windows.graphics.directx.direct3d11.interop.h>
#pragma comment(lib, "d3d11.lib")
#include "hps.h"
#include "sprk.h"
#include "hoops_license.h"
#include "../vr_shared/arvr.h"
using namespace Microsoft::Holographic;
using namespace Microsoft::WRL;
using namespace Microsoft::WRL::Wrappers;
using namespace Windows::Graphics::Holographic;
using namespace Windows::Graphics::DirectX::Direct3D11;
using namespace Windows::Perception::Spatial;
static const int VIDEO_FRAME_WIDTH = 1280;
static const int VIDEO_FRAME_HEIGHT = 720;
static const float window_aspect = (float)VIDEO_FRAME_WIDTH / VIDEO_FRAME_HEIGHT;
static HPS::Canvas canvas;
static HPS::CameraKit camera_kit;
static HPS::OffScreenWindowKey window_key;
static HPS::View view;
static HPS::SegmentKey help_seg;
static Platform::String^ ip_address = L"127.0.0.1";
static Windows::Graphics::Holographic::HolographicSpace^ holographic_space;
static HolographicStreamerHelpers^ streamer_helpers;
static Windows::Perception::Spatial::SpatialStationaryFrameOfReference^ reference_frame;
static bool connected = false;
static HolographicFrame^ Update(Windows::Foundation::Numerics::float3 const & focus_point)
{
if (!holographic_space)
return nullptr;
// The HolographicFrame has information that the app needs in order
// to update and render the current frame. The app begins each new
// frame by calling CreateNextFrame.
HolographicFrame^ holographic_frame = holographic_space->CreateNextFrame();
// Get a prediction of where holographic cameras will be when this frame
// is presented.
HolographicFramePrediction^ prediction = holographic_frame->CurrentPrediction;
auto current_coordinate_system = reference_frame->CoordinateSystem;
for (auto pose : prediction->CameraPoses)
{
HolographicCameraRenderingParameters^ camera_parameters = holographic_frame->GetRenderingParameters(pose);
camera_parameters->SetFocusPoint(current_coordinate_system, focus_point);
}
return holographic_frame;
}
static bool Render(HolographicFrame^ holographic_frame, InitPictureHandler & init_picture_handler)
{
holographic_frame->UpdateCurrentPrediction();
HolographicFramePrediction^ prediction = holographic_frame->CurrentPrediction;
bool atLeastOneCameraRendered = false;
for (auto pose : prediction->CameraPoses)
{
// The projection transform for each frame is provided by the HolographicCameraPose.
auto & cameraProjectionTransform = pose->ProjectionTransform;
// If TryGetViewTransform returns a null pointer, that means the pose and coordinate
// system cannot be understood relative to one another; content cannot be rendered
// in this coordinate system for the duration of the current frame.
// This usually means that positional tracking is not active for the current frame, in
// which case it is possible to use a SpatialLocatorAttachedFrameOfReference to render
// content that is not world-locked instead.
Platform::IBox<HolographicStereoTransform>^ viewTransformContainer = pose->TryGetViewTransform(reference_frame->CoordinateSystem);
if (viewTransformContainer)
{
auto & viewCoordinateSystemTransform = viewTransformContainer->Value;
UpdateCameraParams(
(float const *)&cameraProjectionTransform.Left,
(float const *)&cameraProjectionTransform.Right,
(float const *)&viewCoordinateSystemTransform.Left,
(float const *)&viewCoordinateSystemTransform.Right,
HPS::Window::Driver::DirectX11,
HPS::Drawing::Handedness::Right,
::window_aspect,
init_picture_handler.proj_left,
init_picture_handler.proj_right,
init_picture_handler.view_left,
init_picture_handler.view_right,
::camera_kit);
}
HolographicCameraRenderingParameters^ camera_parameters = holographic_frame->GetRenderingParameters(pose);
ComPtr<ID3D11Resource> resource;
ThrowIfFailed(GetDXGIInterfaceFromObject(camera_parameters->Direct3D11BackBuffer, IID_PPV_ARGS(&resource)));
ComPtr<ID3D11Texture2D> camera_backbuffer;
ThrowIfFailed(resource.As(&camera_backbuffer));
init_picture_handler.target = camera_backbuffer.Get();
window_key.UpdateWithNotifier().Wait();
init_picture_handler.target = nullptr;
// hololens docs say to use previous frame camera for culling.
view.GetSegmentKey().SetCamera(camera_kit);
atLeastOneCameraRendered = true;
}
return atLeastOneCameraRendered;
}
int main(Platform::Array<Platform::String^>^ args)
{
//Initialization of HoloLens. See hololens_remote_1.cpp
// .....
// Main Update/Render loop
for (;;)
{
if (!ProcessWin32Events())
break;
if (connected)
{
//Get current prediction of camera location/rotation
HolographicFrame^ holographic_frame = Update(focus_point);
if (holographic_frame && Render(holographic_frame, init_picture_handler))
{
auto present_result = holographic_frame->PresentUsingCurrentPrediction(HolographicFramePresentWaitBehavior::DoNotWaitForFrameToFinish);
if (present_result == HolographicFramePresentResult::DeviceRemoved)
{
DebugLog(L"Device removed\n");
}
}
}
else
{
window_key.UpdateWithNotifier(HPS::Window::UpdateType::Refresh).Wait();
}
}
init_picture_handler.dx11_device = nullptr;
holographic_space = nullptr;
streamer_helpers->Disconnect();
delete streamer_helpers;
streamer_helpers = nullptr;
HPS::Database::Synchronize();
window_key.UnsetDriverEventHandler(HPS::Object::ClassID<HPS::InitPictureEvent>());
window_key.Delete();
delete world;
return 0;
}
Considerations
Performance is the main concern when using AR. For a smooth experience, 60FPS should be the minimum requirement. Lower or jumpy frame rates can result in the user feeling sick. Because of this, the main render loop should be as bare-bones as possible. If you need to perform expensive calculations, it would be best to perform them on a separate thread.
When in AR mode you should expect that the camera is in constant movement, since it is tied to the position of the AR headset. As a consequence of this, some of the approaches used in highlighting and the default operators will not work in AR mode. For example:
Visualize’s default orbit operator works by modifying the camera position. Since the position of the camera depends on the AR headset, it will be necessary to change the model modelling matrix instead.
Because of the constant camera changes, Visualize will not cache the status of rendering buffers between updates. In turn this means that the overlay highlighting setting of
HPS::Drawing::Overlay::WithZValues, which relies on these cached buffers, cannot be used effectively. Highlights with an overlay setting ofHPS::Drawing::Overlay::WithZValuesin an AR application will be performed sub-optimally.
Currently the Hide operation, and in general Highlight operation with the InPlace Overlay settings are very expensive to use, performance-wise, and should be avoided in VR mode. This is something we are currently addressing for Visualize 2018 SP2.
Some view dependent geometry, like non-transformable text and patterned line will not transform correctly in stereo mode. This will create artifacts where it will look like two instances of said geometry are visible, unless the user closes one of their eyes.
UI design in the HoloLens is an active area of interest. For Microsoft’s best practices, see here: https://docs.microsoft.com/en-us/windows/mixed-reality/design
For developing UI with the HoloLens, you can directly query the HoloLens API. See examples of gesture detection, speech recognition, and more here: https://github.com/Microsoft/MixedRealityCompanionKit
(Remoting) Network latency is paramount to a well-behaved remoting application. It is recommended to use the
HolographicStreamerHelpers->SetMaxBitrate()function to set an appropriate bitrate, to balance image quality with latency. We recommend a value of around 4k.(Remoting) Since the PC’s graphics card is used for rendering, it is recommended that you do not simultaneously run graphics intense processes on the PC as the remoting application.
