Ttgjyuk

$triangle[ABC]$ is a 30-60 right triangle, and its right angle is at C. A is at the origin. A circle is inscribed in it; its center is at

O = (2*sqrt(3)*(sqrt(3) - 1), 2*(sqrt(3) - 1))

and its radius is 12(sqrt(3) - 1). Leg AC is the shorter leg. The equation of the line through it is y = sqrt(3)*x. The line perpendicular to AC has slope -sqrt(3)/3, and the line through O with slope -sqrt(3)/3 is

y = (-sqrt(3)/3)*(x - 2*(sqrt(3))*(sqrt(3)-1)) + 2*(sqrt(3)-1) .

The two lines intersect on leg AC at

Q = (8*sqrt(3)*(sqrt(3)-1), 24*(sqrt(3)-1)) .

So, the command draw (O) -- (Q); should draw a radius of the circle to leg AC. On my computer, the command renders a line segment through the other leg and ridiculously long. It seems to me that the command locating point Q has been ignored.

documentclassamsart
usepackageamsmath



usepackagetikz
usetikzlibrarycalc,intersections


begindocument

noindent hspace*fill
begintikzpicture

path (0,0) coordinate (A) (8,0) coordinate (B) (2,2*sqrt(3)) coordinate (C);
node[anchor=north, inner sep=0, font=footnotesize] at (0,-0.15)textitA;
node[anchor=north, inner sep=0, font=footnotesize] at ($(B) +(0,-0.15)$)textitB;
node[anchor=south, inner sep=0, font=footnotesize] at ($(C) +(0,0.15)$)textitC;
draw (A) -- (B) -- (C) -- cycle;
path let n1=2*(sqrt(3))*(sqrt(3)-1), n2=2*(sqrt(3)-1) in coordinate (O) at (n1,n2);
draw[fill] (O) circle (1.5pt);
draw[blue] let n1=2*(sqrt(3)-1) in (O) circle (n1);


path let n1=2*(sqrt(3))*(sqrt(3)-1) in coordinate (P) at (n1,0);
node[anchor=north, inner sep=0, font=footnotesize] at ($(P) +(0,-0.15)$)textitP;
draw (O) -- (P);
path let n1=8*sqrt(3)*(sqrt(3)-1), n2=24*(sqrt(3)-1) in coordinate (Q) at (n1,n2);
draw[fill=green] (Q) circle (1.5pt);
draw[green] (O) -- (Q);


endtikzpicture

enddocument

I am sorry, I cannot follow your equations at all. you ask TikZ to do

 path let n1=8*sqrt(3)*(sqrt(3)-1), n2=8*3*(sqrt(3)-1) in coordinate (Q) at (n1,n2); 

which is equivalent to

 path (8*sqrt(3)*(sqrt(3)-1),8*3*(sqrt(3)-1)) coordinate (Q); 

(meaning you do not need calc for that), and this is where TikZ places the point. I cannot tell you everything that went wrong in your computation of Q, but here is one point: how is it possible that you do not need the coordinates of O in your way of doing things? You should be solving

 alpha * 1 = O_x + beta
 alpha * sqrt(3) = O_y - beta * sqrt(3)/3 

if you want to find the point where AC intersects with the line that is perpendicular and runs through O, but I cannot see you doing this. (BTW, there is intersection cs: specifically for that, you do not need to do such things by hand.)

Luckily these projections can be done with calc out of the box.

documentclassamsart
usepackageamsmath
usepackagetikz
usetikzlibrarycalc
begindocument
noindent hspace*fill
begintikzpicture
draw (0,0) coordinate[label=below:$scriptstyle A$] (A) --
(8*1,0) coordinate[label=below:$scriptstyle B$] (B) --
(8*(1/4),8*sqrt(3)/4) coordinate[label=above:$scriptstyle A$] (C) -- cycle;

draw[fill] (8*(sqrt(3)/4)*(sqrt(3)-1),8*(1/4)*(sqrt(3)-1)) 
 coordinate (O) circle (1.5pt);
draw[blue] (O) circle(8*(sqrt(3)-1)/4);

path ($(A)!(O)!(C)$) coordinate[label=left:$scriptstyle Q$] (Q)
 ($(A)!(O)!(B)$) coordinate[label=below:$scriptstyle P$] (P);
draw (O) -- (P);
draw[fill=green] (Q) circle (1.5pt);
draw[green] (O) -- (Q);
endtikzpicture
enddocument

Just for fun with tkz-euclide.

documentclassamsart
usepackageamsmath
usepackagetikz,tkz-euclide
usetikzlibrarycalc
usetkzobjall

begindocument
begintikzpicture
coordinate (A) at (0,0);
coordinate (B) at (8,0);
coordinate (C) at (2,2*sqrt(2.99));
tkzDefCircle[in](A,B,C)
tkzGetPointO tkzGetLengthrIN
tkzDrawPoints(O)
tkzDrawCircle[R,color=blue](O,rIN pt)
tkzLabelPoints[below](B)
tkzLabelPoints[above left](A,C)
tkzDrawPolygon(A,B,C)
tkzDefPointBy[projection= onto A--C](O) tkzGetPointQ
tkzDefPointBy[projection= onto A--B](O) tkzGetPointP
draw (O)--(Q) (O)--(P)node[below]$P$;
filldraw [green](Q) circle (1.5pt);
node at (Q)[left]$Q$;
endtikzpicture

enddocument