Let $f(x)={x}^{3}+a{x}^{2}+bx+5{\sin}^{2}x$ be an increasing function on the set $R$. Then, $a$ and $b$ satisfy

$f(x)=x^3+ax^2+bx+5\sin^2x$

$f^{\prime}(x)=3x^2+2ax+b+5.2\sin x\cos x>0$

$f^{\prime}(x)=3x^2+2ax+b+5\sin 2x >0$

$-1\le \sin 2x\le 1$

$\sin 2x=-1$

$f^{\prime}(x)=3x^2+2ax+b+5(-1)=3x^2+2ax+b-5 \ge 0$

$f^{\prime \prime}=6x+2a$

$6x+2a=0$ or, $x=\dfrac{-2a}{6}=\dfrac{-a}{3}$

so now,

$3\left(\dfrac{-a}{3}\right)^2+2a.\left(\dfrac{-a}{3}\right)+b-5\ge 0$

$\dfrac{a^2}{3}-\dfrac{2a^2}{3}+b-5\ge 0$

$\dfrac{-a^2}{3}+b-5\ge 0$

$a^2-3b+15<0$