Okay so you take two cases.
One where 2 boys sit next to each other and the other when all 3 sit next to each other.
Case 1)
From the three boys you choose 2. So 3C2. Then you fix one of the boys positions anywhere. The second boy has 2 spots to choose and sit next to the other guy. Either his left hand side or right. So far, 3C2 x 2. Now two boys are together, the third must not sit beside any of the boys and therefore he only has 4 places to choose from. Then, because there's no restrictions on girls, they can be arranged in 5! ways.
So for two boys next to each other, number of ways = 3C2 x 2 x 4 x 5! = 2880
Case 2) three boys together
Let boys be ABC AND GIRLS be dashes
If you place seats in a line it'll be ABC-----
Bold parts = the 8 seats. The three boys sitting together can be arranged in 3! ways. Then the rest of the girls can be arranged in 5! ways because it's a circle. So total ways of 3 boys together = 3! X 5! = 720
Total number of ways = 3600
Probability = 3600/7! (7! Because without restrictions that's total arrangements)
= 3600/5040
=5/7