The first question most people ask when faced with the notion that “the government shouldn't be paying for thing X”, is: “Well then who will provide thing X?”
The answer in every case is the same: the Free-Market.

Free-market capitalism is a network of free and voluntary exchanges in which producers work, produce, and exchange their products for the products of others through prices voluntarily arrived at.
Murray Rothbard
The Free-Market is simply the term used to describe free and voluntary exchange of goods and labour between individuals. This may seem like a simplistic alternative to government “providing” things to people, but in reality it is the only way wealth can be created without taking it from somewhere else involuntarily.
A Free-Market capitalist may choose to create a product—for the sake of this exercise, let's say wooden crates. The capitalist must purchase the wood to make the crates, purchase the nails to bind them together, and employ the people who will assemble the crates. The capitalist must decide how many crates to make (based on how many he believes he will sell), how much to pay the workers (it must be enough that he attract competent ones, but not so much that he cannot make a profit from the sales), what type of wood to use, how to design the crates, what price to set them at, and where to sell them.

If history could teach us anything, it would be that private property is inextricably linked with civilization.
Ludwig von Mises
The Free-Market capitalist is incurring a great deal of risk in their enterprise, and they are not guaranteed a profit at the end. If they do make a significant profit, such that other capitalists notice there is a fruitful market, they will also have attracted competitors, looking to create wooden crates of either higher quality or lower price (or both if possible).
This is on a basic level how the Free-Market operates. So, who benefits from this arrangement? The capitalist will sometimes benefit in terms of profit, but not always. The greatest beneficiary of a competitive and healthy Free-Market is always the consumer (AKA the average person), who will find they have access to a greater array of higher and higher quality products, at lower and lower prices. The great advancements for the human race which occurred during the Industrial Revolution occurred not because capitalist entrepreneurs earned a lot of money, but because steam engines made creation of high-quality goods much cheaper and quicker. While the average farmer's yearly wage may not have initially increased in the late 1700s, the price of every product imaginable (which only a generation prior had been reserved to royalty and landed gentry) had dramatically dropped thanks to competitive Free-Markets, and was now accessible to them.
The relationship between government and the Free-Market can be described most simply using the terms “public” and “private”. Anything that the government decides to spend tax-payer money to provide for the community becomes part of the “public sector” of trade. Anything that private individuals create and sell to other private individuals for profit is part of the “private sector”.
Once the State, as outlined in the previous section, has established itself as the dominant organisation within a territory, it concerns itself first with justifying its existence. Following this, it concerns itself with expanding its size. Both are achieved by expanding the public sector. It is no surprise therefore that governments around the world are filled with the ever-increasing fervour to claim that more things are “rights”, and must be provided to people for “free”, as they are indispensable services which couldn't possibly be left to “greedy capitalists” to supply.
For reference, if a State economy is entirely comprised of the public sector (that is to say, the State is solely responsible for distributing all goods and services, and has total control of the means of production of a territory) then that State will have adopted the philosophy of Socialism. The 20th Century has taught us, if nothing else, that the quickest way to destroy a country's wealth and the freedom of its populace is for the State to adopt Socialist policies.

Socialism is a philosophy of failure, the creed of ignorance, and the gospel of envy, its inherent virtue is the equal sharing of misery.
Winston Churchill
Libertarians argue that almost everything the government provides would be better provided by the Free-Market. Statists (supporters of the State) argue that certain “rights” cannot possibly be left to the Free-Market, lest poor and disadvantaged people miss out. For the sake of examining the argument, two good examples to analyse are healthcare and cars.
The State in Australia (and almost everywhere in the world) provides a basic level of healthcare for “free”, meaning “at no cost”. Of course, it's not for “free”, because the hospitals, doctors, nurses, and equipment all need to be paid for, and are paid for through taxes and inflation—but for the purposes of making a political slogan, the political class refer to the notion that “healthcare is free”.
There is still the opportunity for private healthcare to exist, however the market is distorted due to the “free” healthcare provided. There is no market available for a capitalist entrepreneur to try to cater to people who have less money and seek a minimally expensive standard of care: because the entrepreneur cannot possibly hope to compete with the “free” healthcare.
This is especially the case since most citizens would already be paying income taxes, and therefore feel they need to take advantage of the services provided by the State.
Due to this, the “free” service provided by the State stagnates. It has no market competition, as nobody could hope to compete against “free” healthcare for low-income earners. Because there is no competition, there is no incentive for the system to maintain a high quality of care and to minimise costs—the healthcare system will always receive the same amount of money from the government regardless of its performance (if anything, a lesser performance will lead to politicians insisting the public health system needs more money). The quality of the service therefore relies entirely on the goodwill and self-imposed diligence of the government employees working in the public health system. These people's pays are also standardised, and therefore there is no financial incentive for them to do a better job than anyone else. Those that take upon themselves the responsibility of ensuring a high standard of care will not necessarily receive financial remuneration.
Because the Free-Market cannot compete for the business of low-income earners who wish to access healthcare services, the private healthcare market only exists for high-income earners. Here, people may pay significantly more money for a higher quality of care, as this is the only way capitalist entrepreneurs can make a profit.
This leads to the political class making the obvious observation: public healthcare is “free”, private healthcare is expensive. The connection they then draw is that without “free” public healthcare, only expensive healthcare would exist and a huge swathe of people would be left without care.
This is absurdly false.
Clearly the first thing that would occur is the market of low-income earners seeking healthcare would be flooded with entrepreneurs seeking to create a low-cost healthcare alternative. The competition between these capitalists would then lead very quickly to better and better care for lower and lower prices, as the entrepreneurs would seek to increase their profits by enticing as many customers as possible, despite earning little profit per each customer.
The notion that the public sector is required to provide basic services at a low cost is an absolute fallacy, and any market that is relatively free shows this. A good example for comparison is the market for cars.
Cars, like healthcare, have a huge spread in prices across the market from cheapest to most expensive. According to the statist logic of healthcare, car manufacturers should be making only exorbitantly expensive cars for the rich. If someone couldn't afford a $200,000 to $300,000 Ferrari or Lamborghini, they would be forced to walk. And yet reality shows us that by far the most competitive market for cars is that of the affordable car around the $20,000 to $40,000 range if purchasing new (there is also a competitive secondary market for used cars). Because there are far more potential customers in this market, the potential profits for a company are far greater, provided they can supply a good quality car for a low price. Ford, Honda, Toyota, and Hyundai are all competing for the same customers, and therefore are in a constant battle to try and provide a better product for a cheaper price. Their profit margins for each vehicle would be far less than the profit margins for a single Ferrari car, but in selling to a higher volume their overall profits are higher.
This is how every single market would work, if it weren't for State interference through the public sector. Imagine now, the government builds a vehicle, and provides it to everyone for “free”. Immediately, Ford, Honda, Hyundai, and Toyota would be pushed out of the market–for how can they compete with “free”, even if it is an inferior product? Ferrari and Lamborghini would be relatively unchanged, for their clientele is made up of people wealthy enough to want something far better than a “free” car, even if they must pay a far higher price. The government's “free” car would not be competing with any other company, and would therefore have no financial repercussions for: breaking down easily, not lasting very long, not going very fast, having poor gas mileage, having poor safety features, or having any other shortcomings.
The creation of the “free” service has distorted the market and caused the average person to have to choose between a “free” and poor quality product, or an exorbitantly expensive and high quality product.
This analysis is true for the vast majority of areas in which the government provides a service, and the services government seeks to provide in Australia are ever-increasing: welfare, construction, transport, healthcare, education, law enforcement, infrastructure, regulation, energy, the judicial system, finance, research, emergency response. All of these areas are less efficient, less productive, and more expensive because they have replaced the “private sector” with the “public sector”, or the Free-Market with the State.