Hamas-linked media said the car his sons were travelling in was hit in Al-Shati camp near Gaza City.
Haniyeh said that the incident would not change Hamas's demands in talks aimed at reaching a ceasefire deal.
Israel's military said the sons were members of Hamas's military wing - a claim denied by Haniyeh.
The group was reportedly on its way to a family celebration to mark the first day of the Muslim holiday of Eid.
Haniyeh told the broadcaster Al Jazeera that three sons - Hazem, Amir, and Muhammad - had remained in Gaza during the war.
A statement from Hamas later said four of Haniyeh's grandchildren - Mona, Amal, Khaled and Razan - were among those killed in what they called the "treacherous and cowardly" strike.
Haniyeh said he heard the news as he was visiting wounded Palestinians who had been taken for treatment to the Qatari capital, Doha, which is where the Hamas leader lives.
"The enemy will be delusional if it thinks that targeting my sons, at the climax of the [ceasefire] negotiations and before the movement sends its response, will push Hamas to change its position," he told Al Jazeera.
In comments reported on Hamas's Telegram channel, he thanked God for the "honour" bestowed on him by what he called the "martyrdom of his children and grandchildren".
Israel's military said it had "eliminated three Hamas military wing operatives in the central Gaza Strip", adding that they were the sons of Ismail Haniyeh. The statement did not mention the reported deaths of Haniyeh's grandchildren.
Speaking to Reuters on Thursday, Haniyeh denied that his sons were Hamas fighters.
As international pressure to reach a ceasefire deal grows, US President Joe Biden has sent the head of the CIA, William Burns, for the latest round of talks in Cairo.
The latest proposal, which Hamas says it is analysing, reportedly includes the release of 40 Israeli hostages being held in Gaza in return for 900 Palestinians from Israeli jails.
Haniyeh is widely considered Hamas's overall leader and has been a prominent member of the movement since 1980. He was elected head of Hamas's political bureau in 2017 and the US Department of State designated him a terrorist in 2018.
These are not the first members of Haniyeh's family to have been killed in the war. Another son was reportedly killed in February, while his brother and nephew were killed in October, followed by a grandson in November.
Hamas-led gunmen attacked Israel on 7 October, killing 1,200 people and taking more than 250 hostage.
Israel says that of 130 hostages still in Gaza, at least 34 are dead.
More than 33,000 Gazans, the majority of them civilians, have been killed during Israel's offensive in Gaza since the October attack, the Hamas-run health ministry says.
By Vicky Wong