Today I did more than two hours of algebra practice. It was fun, except for the sections devoted to simplifying messes of exponents. There are just too many opportunities for arithmetic errors in problems like that.
I did run into a bit of confusion in another section regarding one of the answers provided in the review materials, but I got help at the Math Help Forum, which is a good resource for those self-studying.
I don’t think I’m happy with any of the answers you’ve gotten. Let’s just look at the first problem, to simplify
$$
\sqrt{xy^{\vphantom3}}\sqrt{x^3y}.
$$
I’m assuming all this is happening in $\mathbb R$, not $\mathbb C$.
Then the first square root is defined only if $xy\ge0$. Same thing for the second square root. When you work out all the possibilities,
$$
\sqrt{xy^{\vphantom2}}\sqrt{x^3y}=
\begin{cases}
x^2y,\quad&\text{if $xy\ge0$ and $y\ge0$},\\
-x^2y,\quad&\text{if $xy\ge0$ and $y<0$},\\
\text{undefined,}\quad&\text{if $x<0$ and $y>0$},\\
\text{undefined,}\quad&\text{if $x>0$ and $y<0$}.\\
\end{cases}
$$
Whatever answer we give has to include the fact that half the time, $\sqrt{x^{\vphantom2}y}\sqrt{x^3y}$ doesn't make sense at all. So nothing that just has $x$ and $y$ and absolute values can be correct. We could simplify it as $|x|(\sqrt{xy})^2$, but I think any answer we give has to involve either square roots or a bunch of cases.
Does this make sense? It's certainly possible that I'm confused.
I’ve corrected the typesetting issue. (The fact that comments can’t be edited by their authors is a problem for a blog like this.) I don’t have energy to engage properly with the question tonight, but I’ll try to get back to it at the beginning of my study time tomorrow.
I think you are right. My naive answer of $\sqrt{xy}\sqrt{x^3y}=x^2|y|$ is the one given in the book, however, which raises the issue of what the book actually meant by its question. After some investigation, I think it must have been asking for the answer when the original expression is defined in $\mathbb{R}$. (This encompasses your first two cases, which both have a solution of $x^2|y|$, since when $y<0$, $|y|=-y$.) In that light, the answer I got on the Math Help Forum is correct as far as it goes.
I considered whether the question was asking for a solution that would be true even if the original expression was defined in $\mathbb{C}$ only, but when I worked it out, your second two cases turned out to have a solution of $-x^2|y|$.
When $x<0$ and $y>0$,
$$\sqrt{xy}\sqrt{x^3y}=\sqrt{-1|x|y}\sqrt{-1|x^3|y}=i\sqrt{|x|y}\left(i\sqrt{|x|^3y}\right)=i^2\sqrt{|x|y}\sqrt{|x|^3y}=-1\sqrt{|x|^4y^2}=-|x|^2|y|=-x^2|y|\text{.}$$
When $x>0$ and $y<0$,
$$\sqrt{xy}\sqrt{x^3y}=\sqrt{-1x|y|}\sqrt{-1x^3|y|}=i^2\sqrt{x|y|}\sqrt{x^3|y|}=-1\sqrt{x^4|y|^2}=-x^2|y|\text{.}$$
This has been a good reminder to be careful with square roots.
(Also, I found this useful website for checking one's math code before posting: https://quicklatex.com/)