It’s tucked away so you can’t see it, and working toward building up wealth for your retirement. That’s what a 401(k) retirement fund is supposed to be doing for you.

But suddenly, you get the idea to borrow from it to buy a home. It’s legal. It’s allowed by your employer. But many financial experts warn against doing it at all.

“Taking money out of your 401(K) should be the last, last resort. People make the argument that they are paying themselves back, but that doesn’t always happen,” says Nicole Middendorf, CEO and financial advisor  at Prosperwell Financial in Minneapolis.  She also is author of the books Lipstick on the Piggy Bank and Simple Answers: Life is More Than Just about Money.

Middendorf worries when someone borrows from their 401(k) because they usually are not seeking out an expert’s opinion about all the different options and ramifications. They just see that they have a certain amount in their retirement account and find out they can take out 50 percent of the value.

“Buying a house isn’t necessarily an investment. Real estate historically appreciates 3-4 percent a year. That’s inflation. There’s nothing wrong with renting for a while until you can accumulate enough money to have a down payment,” she says.

She has seen people take money out of their retirement plan to buy a house, and then they lose their job. That money they took out becomes categorized as a distribution, and they end up paying a 10 percent penalty plus federal taxes.

“That’s the worst case scenario,” she says.

Check today’s mortgage rates here.

If someone does borrow from their 401(k), the loan does not get reported to the credit agencies, and they are paying back the interest to themselves. Plus, the interest rate is usually lower than if they had to go someplace else to get a loan. These are all good things.

“But when you take the money out of your retirement fund, you are taking that money out of the market,” Middendorf says. “You are taking money away from yourself for the long term.”

Most people she meets don’t like to talk about money. And most of them, especially those in their 20s and 30s, think retirement is so far away that they can make up for any withdrawals they take from their 401(k). But that usually doesn’t happen.

Historically, your money should double every 7-10 years in a retirement fund, she says. If you had $100,000, it should accumulate to $200,000 in that time period.

“That’s the power of compounding,” she says.

But if you take out $50,000 out that original $100,000 for a house that means you already stilted the growth of the fund in the market at that point.

For a few scenarios, Middendorf can see taking money out of a 401(k) as a plausible option. For instance in case of a divorce and the couple selling their house, one of them is buying another house but needs to get a down payment out of their retirement fund quickly.

“But they know they will have that money back in 60 days from the sale of their other home, and they will be able to put it back into their retirement fund. It’s rare that all the stars align, and things work out like that,” she says.

For those who have the income to support taking money out of their 401(k) plus the future earnings to put it back, then great, says Darius Jenkins, loan officer at McLean Mortgage Corporation in Fairfax, Va.

“But each 401(k) is different. Some employers do allow you take out money as a first-time homebuyer, and some don’t. You really have to check with your own situation and your own plan,” he says.

Get a personalized rate quote here.

He has seen people take a down payment out of their 401(k) because they didn’t have any other outlet for cash.

“And I’ve seen people do it because they wanted more house,” he says.

It doesn’t even matter what age. People in their late 20s to well into their 50s are taking money from their retirement funds to buy a house, Jenkins says.

“The younger people want the right house right out of the gate instead of the philosophy that the first house gets you into the second house which gets you into the third house down the road. They want it all right off the bat. So they go into their 401 (k) retirement plans,” he says.

He also has seen people who don’t have much of a retirement plan in place at all age levels.

“All they can think about is the house. They don’t see the house as part of their retirement portfolio,” Jenkins says. “Most people should sit down with a financial planner, but I don’t think most people do.”