[กิตติ]

$(ax^2+by^2)(x+y)=7(x+y)$
$ax^3+by^3+xy(ax+by)=7(x+y)$
$16+3xy=7(x+y)$
$16=7(x+y)-3xy$.........(1)

$(ax^3+by^3)(x+y)=16(x+y)$
$ax^4+by^4+xy(ax^2+by^2)=16(x+y)$
$42+7xy=16(x+y)$
$42=16(x+y)-7xy$...........(2)

$(ax^4+by^4)(x+y)=42(x+y)$
$ax^5+by^5+xy(ax^3+by^3)=42(x+y)$
$ax^5+by^5=42(x+y)-16xy$

(1)คูณด้วย 6 $96=42(x+y)-18xy$
$42(x+y)-16xy=96+2xy$
(1)คูณด้วย 16 $16^2=16\times 7(x+y)-48xy$.......(3)
(2)คูณด้วย 7 $6\times 7^2=16\times 7(x+y)-49xy$.........(4)
(3)-(4) $16^2-6\times 7^2=xy$
$2xy=2(16^2-6\times 7^2)$
$=2(256-294)=-76$
$ax^5+by^5=42(x+y)-16xy=96-76=20$........ผิดตรงบรรทัดท้าย

คำตอบที่ถูกคือ $ax^5+by^5=42(x+y)-16xy=96-(-76)=172$