While all the usual dense materials that applied to the bridge also work well for the nut, bone, ebony, rosewood, corian, brass, etc...
I'll start by championing the "no nut" or "zero fret" approach.
What's known as the "zero" fret - a fret placed where the nut would normally be, can circumvent a lot of issues regarding action and set up. Then how you guide the string to your tuner is less critical. you can even have a carved out slot or a saw kerf that makes the string track to the tuner. As long as there is enough angle to the string as it leaves the "zero" fret, buzz should be no problem. Here's a sample on one of my octave dulcimers.
Now sometimes a kind of string spacer nut is used with the zero fret just to keep the strings in correct alignment - If I do that I use a piece of 1/8" Delrin [if you sand the black stuff down to 600 grit and steel wool is with 0000 it looks a lot like ebony] as it adds almost no friction to the tuning process. Another upside of minimal friction besides ease of tuning is you can push on the strings behind the zero fret and get a nice bending sound on your open strings almost like a wammy bar.
Also if your path to your tuner is at a slight angle, rather than a straight line, you can have the string turn around [meaning be slightly deflected by] a guide pin that maintains it's side to side position. This can be anything, but can also be Delrin to reduce friction.
[I know the traditional threaded bolts for nuts give you many choices of which groove to put a string in, but you're still stuck with hoping the groove you made for it results in the right height for the string.]
Whatever we choose to use here, the string wants a fairly clean break point with a back angle to the tuner. If it rests in the bottom of a filed groove or if by accident the groove is higher at the back edge of the nut - the edge towards the tuners - a pesky buzz or whine can arise. This can be interestingly sitar-ish but mostly un-welcome.