This isn't meant to be an all inclusive listing .... but will get you off to a decent start at least. For links to some of the mentioned sources, click on the associated button listed at the left under Support Services.
There are several programs in the various branches of government, including small business loans, contracting, and of course direct non-dilutive investment via SBIR and STTR grant programs.
The Small Business Administration manages many of these programs.
For example, agencies within DOD have a wealth of information and support for small businesses seeking government contracts.
The NIH and other granting agencies have legislated budget set-asides specifically for their SBIR programs and, although not part of the budget set-aside, the STTR programs also facilitate start-ups.
Another legislative example is the Bayh-Dole Act, that allows non-profits such as universities to own intellectual property (IP) generated off of government research grants. While not specifically in support of start-ups, the B-D Act helps new company formation out of univerisities and other non-profit reseaerch institutes based on grant-funded IP. The fees for filing patents with the USPTO are lower for small entitites than large companies too.
The list goes on.....