Recently the method of smoothed particle hydrodynamics developed rapidly and was already used for the solution of a wide spectrum of scientificengineering problems in the fields of astrophysics, hydrodynamics, gasdynamics and so on. Besides, there were solved many problems on liquid motion in a ves-sel with an open wall. But at the solution of specific problems typical for the field of metallurgy, simulators created with the aid of the method shown may behave unstably. Models instability is characterized with an uneven distribution of liquid in a vessel and particles penetration through rigid walls and with the isolation of particles from a basic mass of liquid. The particles isolation from the mass of liquid and the entrainment of surrounding particles with it results in non-physical behavior of liquid and, as a consequence, in wrong results of computation. This paper reports the analysis of the applica-tion of nucleus different functions to the computation of a density value for the problem solution of particle isolation from the basic mass of liquid and their entrainment of surrounding particles with them. For re-sults check a model of liquid steel oscillation in a steelladle on the basis of the method of particles smoothing was used.