The "Pilot" is the Key to the Counterbore cutting well. It constrains counterbore in the radial axis, normally counterbore's are used add recess's for screw heads (sit flush with surface) or to spot face surfaces (on casting) for good bearing of the underside of screw heads. A .001-.003 clearance of hole to pilot works well, more clearance (.005-.010) will also work , the difference will be the finished diameter (oversize) and taper of the counterbore. for most uses (screw heads) this is not a issue. For the anchor connectors I would assume that a minimum taper and on size hole would be desirable.
If I was to do this on a drill press I would use a Split Point drill (self centering properties - improved hole location)and not a Chisel point drill which is typical of hardware store drills. Home Depot sells Split Point drill's. the difference is the grind on the end of the drill. drill hole .010-.015 undersize, follow with a Reamer .001 oversize of the pilot diameter. Run the Counterbore at a low speed to reduce chatter (vibration) use WD-40 as a cutting lubricant (works well on aluminum) keeps the chips from gumming up the works, better hole accuracy and finish, with drill press off, insert pilot into hole, add a clamp (C-clamp?) to hold the 80/20 to the drill press again to reduce chatter (vibration)
Split Point drill are also good when drill harder material, Regular (chisel point) drills work harden the material (make the material even harder) Stainless steel in notorious for work hardening when drilling. Regular (chisel point) drills are good for drilling wood.
I am a machinist with access to a Bridgeport manual mill also a CNC mill, so I have no need to do this on a drill press. But I am confident that if the above is followed, satisfactory hole geometry for the "anchor connectors" is possible on a drill press.
Regarding using End mills in a drill press (DONT) Milling machine Spindles are constructed much more accurately and stiffer (preloaded bearing) than drill press spindles, Drills and Countersinks with their tapered points have a dampening and stabilizing effect on the tool also self centering effect when drilling into solid surface's. ever notice how a drill chatters when opening up an existing hole to a larger diameter.
End mill's with a square end will exercise all the play in the drill press spindle when drilling into solid material, combined with the inherent lack of stiffness in most drill presses and you got a hot mess.
While it may seem like a lot of work it goes quick, if one was doing thousands of holes I wouldn't recommend this method, the average DIY van builder is maybe going to need maybe 50 anchor connectors? at most.
Once the pilot hole is drilled the Reamer and Counterbore will follow the existing hole (no choice) A drill fixture (with a hardened drill bushing) to locate the pilot hole would make it almost foolproof.