Optitrak + Projector touch table/surface

Calibration with Tracking Tools

note: if you are using the laptop setup in room 3108, make sure that the dongle for Tracking Tools is not plugged in when starting up the computer.

To calibrate the optitrak for use with the touch table, make sure that the Tracking tools dongle is plugged in. Then start Tracking Tools from the desktop.

In general, you will follow these steps in order to calibrate:

1. Use the wand to register the area of interest
2. Use the plane to set the ground plane
3. If using trackables, create trackable objects
4. Set the data to stream out (for reading into matlab)

Step-by-step calibration guide

1. Open Tracking Tools and select "Perform Calibration"

2. Change Illumination Type from "Strobe IR" to "Continuous"

3. Clear the table of the reflective equipment (e.g., finger ball, wand, ground plane) and make sure that there are no bright lights registered in the software. If there are, find their source on or around the table and remove them so that the cameras images are all dark. Once this is the case move on to step 4.

As a last resort if there are any lights visible in the cameras' videos that can't be removed, you can block out that area in the corresponding camera with a mask. To do so, click the "Block Visible Markers" button, and it will automatically mask any markers currently registered by the cameras (as indicated by a red mark corresponding to the blocked area in the camera's field). You can use the "Clear Blocking" button to remove masks if you need to adjust or start again. Once all lights are masked, move on to step 4.

4. Click on "Start Wanding" [(1) in the picture below] and sweep the wand over the table's surface. Your goal is to a) wand the 3D area you want to track in, e.g., most likely the area close to the table's surface; and b) "paint" all the regions as full of wand data as possible. e.g., in the picture below, camera 4 is more painted than camera 5.

When the indicated quality becomes "Very High" (2), you can click on "Calculate" (3).

5. The tracking tools software will now calculate the quality of your calibration. If you wanded well, you should ultimately not see the distorted square grid shown in Cameras 2 and 3 below. Bear in mind that the grid will become less distorted as the results of the calibration are being calculated, however if the distortion in the grid persists when the cameras read "Exceptional", you should calibrate again. When all cameras achieve "Exceptional" results, and the overall result is "Exceptional", and there is no distortion in the grids for any of the cameras, click on "Apply Result" (see picture below).

Click on "Apply" again when the dialogue pops up, and save the results of your calibration (by default the file name will include the date and time- you can load the same calibration later in the same session if you are sure that the setup has not changed).

6. Set the ground plane on the lower left corner of the table top so that Z arrow points towards the top of the projected image. Click on "Set Ground Plane", and save the results.

7. Now stream out the data so that it can be read into Matlab. To do so, click on the "Streaming Pane" icon. Put a check in the "Broadcast Frame Data" box, and change Type from "Multicast" to "Unicast". You can now minimize Tracking Tools and it will output data continuously in the background.

Matlab scripts

Function: Setting the optitrak plane in matlab

In place of the matlab code for reading in optitrak data, the following matlab function can be used to:

1. read in data sent from the optitrak system, and furthermore
2. set the table top plane, as understood by the Optitrak, into variables for reference in matlab, and
3. transform the native coordinate system of the optitrak (x z y) into another, possibly more familiar coordinate system (x y z).

This function will return:

This function should be called before opening a main window with psychtoolbox.

function [touch_plane_info, client] = opti_setup_touch_plane_srt

%This gives an error (freezes) if the NET assembly is already loaded.  In
%that case you need to quit Matlab and start over

if ~IsAssemblyAdded('NatNetML')  
	Opti = NET.addAssembly('C:\Users\Jim Enns\Documents\MATLAB\NatNetSDK2.2\NatNetSDK\lib\NatNetML.dll');
end

client = NatNetML.NatNetClientML(1);
%client.Initialize('142.103.0.167','142.103.0.167');
client.Initialize('142.103.251.175','142.103.251.175');
data = client.GetLastFrameOfData;
	

reply = 'N';
while ~strcmpi(reply,'Y')
	
	answer = {};
	while isempty(answer)        
		prompt={'Pixel Coords: '};
		name='Lower Left Corner';
		numlines=[1,35]; %this width allows for title to be seen
		defaultanswer={'[0 768]'};
		answer=inputdlg(prompt,name,numlines,defaultanswer);

		%convert from string to number data for output variables
		if ~isempty(answer)
			OriginPixelCoords = str2num(answer{1});
		end
	end

	pDataDest = [data.OtherMarkers(1).x data.OtherMarkers(1).z data.OtherMarkers(1).y];
	
	Question = {'LOWER LEFT pre coordinate transform:'...
		...
		'Marker 1: ' num2str(pDataDest(1:3))...
		...
		'Accept?'...
		};
	ButtonName = questdlg(Question, 'Data Check', 'Yes', 'No', 'Quit', 'Yes');

	switch ButtonName
		case 'Yes'
			orig_pos = pDataDest;
			reply = 'Y';
		case 'No'
			reply = 'N';
		case 'Quit'
			%quit opto too?
			%need to pass quit flag so this actually works?
			transform_info = [];
			clear all; close all; Screen('CloseAll'); 
			disp('*** Exiting Program ***');
			return
	end

end

%take a point on +ve x axis
reply='N';
while ~strcmpi(reply,'Y')

	answer = {};
	while isempty(answer)        
		prompt={'Pixel Coords: '};
		name='Lower Right Corner';
		numlines=[1,35]; %this width allows for title to be seen
		defaultanswer={'[1024 768]'};
		answer=inputdlg(prompt,name,numlines,defaultanswer);

		%convert from string to number data for output variables
		if ~isempty(answer)
			XAxisPixelCoords = str2num(answer{1});
		end
	end


	 pDataDest = [data.OtherMarkers(1).x data.OtherMarkers(1).z data.OtherMarkers(1).y];
	
	Question = {'LOWER RIGHT pre coordinate transform:'...
		...
		'Marker 1: ' num2str(pDataDest(1:3))...
		...
		'Accept?'...
		};
	ButtonName = questdlg(Question, 'Data Check', 'Yes', 'No', 'Quit', 'Yes');

	switch ButtonName
		case 'Yes'
			x_axis = pDataDest;
			reply = 'Y';
		case 'No'
			reply = 'N';
		case 'Quit'
			%quit opto too?
			%need to pass quit flag so this actually works?
			clear all; close all; Screen('CloseAll'); 
			disp('*** Exiting Program ***');
			return
	end
end

% point on the xy plane
reply='N';
while ~strcmpi(reply,'Y')

	answer = {};
	while isempty(answer)        
		prompt={'Pixel Coords: '};
		name='Upper Left Corner';
		numlines=[1,35]; %this width allows for title to be seen
		defaultanswer={'[0 0]'};
		answer=inputdlg(prompt,name,numlines,defaultanswer);

		%convert from string to number data for output variables
		if ~isempty(answer)
			YAxisPixelCoords = str2num(answer{1});
		end
	end

	 pDataDest = [data.OtherMarkers(1).x data.OtherMarkers(1).z data.OtherMarkers(1).y];
	
	Question = {'UPPER LEFT pre coordinate transform:'...
		...
		'Marker 1: ' num2str(pDataDest(1:3))...
		...
		'Accept?'...
		};
	ButtonName = questdlg(Question, 'Data Check', 'Yes', 'No', 'Quit', 'Yes');

	switch ButtonName
		case 'Yes'
			y_axis = pDataDest;
			reply = 'Y';
		case 'No'
			reply = 'N';
		case 'Quit'
			%quit opto too?
			%need to pass quit flag so this actually works?
			clear all; close all; Screen('CloseAll'); 
			disp('*** Exiting Program ***');
			return
	end
end

% finish the screen rect (event though this is redundant for establishing
% the plane)
reply='N';
while ~strcmpi(reply,'Y')

	answer = {};
	while isempty(answer)        
		prompt={'Pixel Coords: '};
		name='Upper Right Corner';
		numlines=[1,35]; %this width allows for title to be seen
		defaultanswer={'[1024 0]'};
		answer=inputdlg(prompt,name,numlines,defaultanswer);

		%convert from string to number data for output variables
		if ~isempty(answer)
			OppCornerPixelCoords = str2num(answer{1});
		end
	end

	 pDataDest = [data.OtherMarkers(1).x data.OtherMarkers(1).z data.OtherMarkers(1).y];
	
	Question = {'UPPER RIGHT pre coordinate transform:'...
		...
		'Marker 1: ' num2str(pDataDest(1:3))...
		...
		'Accept?'...
		};
	ButtonName = questdlg(Question, 'Data Check', 'Yes', 'No', 'Quit', 'Yes');

	switch ButtonName
		case 'Yes'
			opp_corner = pDataDest;
			reply = 'Y';
		case 'No'
			reply = 'N';
		case 'Quit'
			%quit opto too?
			%need to pass quit flag so this actually works?
			clear all; close all; Screen('CloseAll'); 
			disp('*** Exiting Program ***');
			return
	end
end

%Create the Coordinate Systesm
[T_opto_plane,T_plane_opto]=MakeCoordSystem(orig_pos,x_axis,y_axis);

%Display the new transformed coordinate system
disp('Points in Local Coordinate System are:')
[new_orig]=transform4(T_opto_plane, orig_pos)
[new_x_axis]=transform4(T_opto_plane, x_axis)
[new_y_axis]=transform4(T_opto_plane, y_axis)
[new_opp_corner]=transform4(T_opto_plane, opp_corner)

touch_plane_info.T_opto_plane = double(T_opto_plane);
touch_plane_info.T_plane_opto = double(T_plane_opto); 
touch_plane_info.opto_rect = double([new_orig; new_x_axis; new_y_axis; new_opp_corner]);
touch_plane_info.pixel_rect = double([OriginPixelCoords; XAxisPixelCoords; YAxisPixelCoords; OppCornerPixelCoords]);
touch_plane_info.old_opto_rect = double([orig_pos; x_axis; y_axis; opp_corner]);

%calculate the pixel to opto conversion
mm1 = double((new_x_axis(1) - new_orig(1))/(XAxisPixelCoords(1) - OriginPixelCoords(1)) * 1000);
mm2 = double((new_opp_corner(1) - new_y_axis(1))/(OppCornerPixelCoords(1) - YAxisPixelCoords(1)) * 1000);
mm3 = double(-(new_orig(2) - new_y_axis(2))/(OriginPixelCoords(2) - YAxisPixelCoords(2)) * 1000);
mm4 = double(-(new_x_axis(2) - new_opp_corner(2))/(XAxisPixelCoords(2) - OppCornerPixelCoords(2)) * 1000);
touch_plane_info.mmPerPixel = mean ([mm1 mm2 mm3 mm4]);